AI benchmark relevance registry

Source of truth for which benchmarks are relevant right now, and how much weight each carries when ranking models. Consumed by:

• services/ai/curator (Loop C) — proposes additions/relevance changes from

  the vetted allowlist + an LLM/web scan. The scan only enqueues candidates (see Candidate queue); a benchmark enters the decision weighting only after internal-staff vetting (contamination / methodology / trust).

• services/ai/eval — selects which suites to run and how to weight scores

  into per-task rankings; the tiered runner (stack-RFC-008 D4) reads tier.

• services/ai/curator (Loop A) — combines modelreg benchmark rows with the

  relevance weights here to rank the served set.

Why this exists. Rankings that decide which models Koder AI serves are only as good as the benchmarks behind them. Suites saturate (everyone scores ~100%), get contaminated (leak into training data), or get superseded by better ones (SWE-Bench Pro, BrowseComp, MCP Atlas all emerged recently). This registry keeps the decision basis current and auditable rather than frozen.

Machine-readable canonical — ai-benchmarks.json

This .md is the human view. The single source of truth consumed by code is the sibling ai-benchmarks.json — one row per benchmark/suite with relevance, weight, tier, trust, modality.

Because the two Go consumers live in separate module roots (they cannot share a go:embed), each vendors a byte-identical copy of the canonical and reads the columns it needs:

• services/ai/eval/internal/benchtier/ai-benchmarks.json → projects tier

  (tiered suite selection).

• services/ai/curator/backend/internal/benchreg/ai-benchmarks.json → projects

  relevance/weight (Loop A ranking; a row with weight > 0 is a ranking benchmark, others are runnable-only suites present for their tier).

Drift between the copies and the canonical is a CI failure, not a silent bug: audit-ai-benchmarks.sh (workflow .gitea/workflows/audit-ai-benchmarks.yml) cmps all three. To edit the registry, change ai-benchmarks.json, then cp it over both vendored copies (the guard prints the exact command on drift). Loop C (CURATOR-006) regenerates the canonical as the vetted set changes. Architectural decision: stack-RFC-008 + resolved EVAL-013 remainder (chose vendored-copy + CI guard over a runtime-loaded file or a shared module — keeps binaries self-contained, no new module edge).

Schema

Allowlist — vetted sources (Loop C backbone)

The polled, trusted sources whose published results feed the ranking deterministically. New sources are added by internal staff, not by the scan.

Benchmarks — vetted (in decision weighting)

Candidate queue — proposed by Loop C scan, awaiting staff vetting

Benchmarks the LLM/web scan surfaced that are not yet in the decision weighting. Promotion to vetted requires contamination + methodology + trust review by internal staff (control-plane). Nothing here affects rankings.

Deprecated / saturated (kept for audit)