autonomous model curation, multimodal discovery and benchmark-relevance tracking

Status: Draft. Records the architecture conversation of 2026-06-01 that started from "the MiniMax M3 in a YouTube video is not in our local base — how does runtime/models.yaml even get populated?" and converged on an autonomous curation subsystem governed by a human control-plane approval. The decision set (D1–D4 below) was deliberated via /k-arch and ratified a/a/a/a by the owner in the same session.

1. Problem

The Koder AI local model base is populated by hand. services/ai/runtime/models.yaml is a hand-curated catalog; runtime/internal/updater (weekly systemd timer) only refreshes the SHA of weights for models already listed and enabled. Which models enter the catalog is 100% human; the assisted-discovery path (updater --discover) scans only text-generation on HuggingFace and merely logs candidates. Consequence: new frontier models (e.g. MiniMax M3, released 2026-06-01) are invisible to the Stack until a human notices, reads, and edits the YAML — and the benchmark set we rank models with is never itself reviewed for relevance, so rankings can silently rest on saturated/contaminated suites.

The owner wants three new capabilities:

• (A) Auto-population/update of the served catalog from benchmark-relevance-weighted scores.
• (B) Periodic multimodal discovery (language, image, video, audio, embedding, …) with automatic analysis of each candidate.
• (C) Benchmark-of-benchmarks meta-routine that researches new benchmarks and re-assesses whether the suite we rank with is still the most relevant available.

2. Thesis — wide autonomous funnel, narrow governed commit

Every capability above shares one shape: discovery and analysis are fully automated; the act of putting a model into production (or a benchmark into the decision weighting) is governed. The funnel is wide and autonomous (discover → measure → rank → propose); the commit is narrow and gated (human approval before serving). This shape recurs in the Stack already (KVS#202 opt-in→default, FLOW-205 CSP flip, desk#172 release management) — it is the "the Stack needs role X to approve action Y" pattern, and its home is services/foundation/{desk,bpm}, not a bespoke inbox.

3. Ratified decisions

D1 — Autonomy: autonomous up to proposal; governed flip (control-plane)

The pipeline discover → eval → rank → propose is automatic. The flip to serving (set enabled in runtime/models.yaml, promote a gateway alias) requires human approval. Crucially this approval is internal-Koder control-plane — a provider/operator decision by Koder's own staff — not a decision exposed to customer tenants (workspace owners/admins on subscription plans). It is scoped to GlobalTenant = "koder" (already encoded in modelreg/types.go: the curated catalog is platform-owned, all-tenant-visible).

• Approval surface: the operator (koder) tenant instance of

  services/foundation/desk approval-workflows (desk#063: approvers by role/hierarchy, SLA, auto-escalation, immutable audit). desk being a multi-tenant SaaS is the isolation mechanism — the Koder-internal org is one tenant, fully separated from customer tenants.

• Roles: internal staff roles in services/foundation/id (Koder ID).
• Notification: infra/observe/notify (koder-notify) fan-out

  (EmailSlackTelegramPagerDutyTeams/Webhook) per role channel.

• Bootstrap adapter: until desk#063 ships, the proposal queue is surfaced

  via the existing reminders.md/notices session hook — demoted to *one pluggable notification adapter* behind the same proposal contract. The curator emits a stable PromotionProposal; the delivery+approval layer is swappable (D3 evolvability / D9 reversibility).

• Does not loosen the ratified modelreg-RFC-001 Q2 gate (koder-curated

  = 2 curator approvals + passing benchmark). Auto-promotion under guardrails is a future destination via an RFC amendment once the loop has a track record (per self-hosted-first.kmd: thresholds cannot be loosened without amendment) — the opt-in→default evidence pattern, not a day-1 default.

D2 — Placement: a new orchestrator sector

A new sector services/ai/curator (working slug — ratify against specs/naming/forms.kmd; alternatives scout, modelscout) owns the active loops. Clean responsibility split:

The curator reuses modelreg #003#004#005 rather than duplicating storage (reuse-first). Folding the loops into modelreg would violate its ratified metadata-vs-executor boundary (D2 coupling); repurposing zoo would conflate the public HF-like hub with an internal scout daemon (D1 wrong abstraction).

zoo boundary & synergy: services/ai/zoo stays as-is here (landing-only, public HF-category "ModelDatasetSpaces Hub") — not built in this work. But it is a planned downstream product whose catalog is seeded by the curator: the same discovery + license-classification pipeline (Loop B + the redistribution gate, curator#010) that feeds Koder AI's internal serving also feeds zoo's public catalog. zoo therefore (a) reads model metadata from modelreg (system of record — never a second store), and (b) mirrors only can_mirror=true weights (license-cleared; gated/NC are referenced, not hosted). The launch cold-start seed strategy is a follow-up ticket in the koder-zoo submodule (zoo is a separate repo); weight hosting at scale rides the object-storage plane (stack-RFC-006). Serving rights (can_serve) and redistribution rights (can_mirror) are classified separately (curator#010).

D3 — Benchmark sourcing (Loop C): hybrid

Backbone = an allowlist of vetted sources (LMArena, HELM, Artificial Analysis, OpenLLM Leaderboard, SWE-bench/Pro, Papers-with-Code, …) that feeds the decision ranking deterministically. On top, a periodic LLM/web scan only enqueues candidate benchmarks into a vetting queue (contamination / methodology / trust review) before any benchmark enters the decision weighting. The adversarial web is never auto-weighted (D6 data integrity / D7 security). Same shape as D1: autonomous discovery, governed promotion.

D4 — Eval compute: tiered

Cheap triage first (ingest published leaderboard scores + a small mini-suite) ranks the bulk; only finalists run the full self-hosted suite (incl. eval kode-private) on the s.khost1 VM per heavy-work-isolation. Bounds GPU/energy (D4 efficiency / hyperscale-first) while preserving our own measurement on the models we would actually serve.

4. Architecture

        LOOP B (discovery)          LOOP C (meta-benchmark)
              │                            │
              ▼                            ▼
   HF multimodal + vetted          allowlist (vetted) + LLM scan
     sources (pipeline_tags)        → ai-benchmarks registry
              │                       (relevance/freshness/trust)
              ▼                            │
   ┌────────────────────────────────────────────────┐
   │  services/ai/curator (orchestrator daemon)      │
   │  LOOP A: rank by benchmark relevance → PROPOSAL │
   └────────────────────────────────────────────────┘
        │ measure (tiered)   │ record                │ propose flip
        ▼                    ▼                       ▼
   services/ai/eval     services/ai/modelreg    desk approval-workflow
   (suite + nightly)    (GlobalTenant)          (internal Koder staff,
                             │                   koder-notify, Koder ID)
                             ▼                       │ approve
                   runtime/models.yaml + gateway alias / ai-model-recommendations.md

Planes. Control-plane (internal Koder staff: curation + approval) × data-plane (customer tenants read the global catalog). A *tenant model preference* (a customer restricting which curated models their workspace uses) is a distinct data-plane axis, out of scope here.

5. The three loops

• Loop A — benchmark-relevance curation. Combine modelreg benchmark rows

  (#004) with the ai-benchmarks relevance weights (Loop C) into a per-task ranking; diff against the current served pick; emit PromotionProposal (enable / disable / promote / demote) → governed flip (D1).

• Loop B — multimodal discovery. Generalize updater.DiscoverNew (today

  text-only, log-only) to all modalities via HF pipeline_tags + vetted sources; auto-analyze each candidate (metadata → modelreg draft; enqueue for tiered eval).

• Loop C — benchmark-of-benchmarks. Maintain registries/ai-benchmarks.md

  (source, modality, relevance, freshness, trust, contamination status); hybrid sourcing (D3); re-weight the eval suite selection from it.

6. Self-hosted-first analysis (5 gates)

The curator substitutes the manual-curation workflow + the text-only --discover; external analogs are managed-MLOps model-selection services.

• G1 feature parity: the manual flow today; curator supersedes it.
• G2 performance: N/A (orchestration, not hot-path).
• G3 stability: bootstrapping.
• G4 capability: scheduler + HTTP polling + eval trigger + proposal queue.
• G5 critical-path: unblocks autonomous, benchmark-grounded model decisions

  for runtime + gateway.

7. Open questions

• Q1: Sector slug — curator vs scout vs modelscout (ratify via forms.kmd).
• Q2: Cadence per loop (B daily? C weekly? A on every eval ingest?) — default

  proposed: B daily, A on benchmark-threshold-cross event, C weekly.

• Q3: PromotionProposal — extend modelreg's tag-proposal (#003) in modelreg,

  or own it in curator and call modelreg only for the resulting tag? Lean: proposal lives in curator (it spans models.yaml + alias, beyond a tag); modelreg records the resulting curated tag.

• Q4: Discovery sources beyond HF for non-text modalities (Civitai for image,

  etc.) — allowlist TBD in Loop B ticket.

8. Scope boundaries (not in this RFC's work)

• Building zoo's backend (only the boundary is recorded).
• Building desk#063 / bpm (design against the contract; bootstrap adapter meanwhile).
• Tenant model-preference (data-plane, future, separate).
• Auto-promotion without a human (future, by amendment).

8.3 — Integração do Índice KVIQ no Roteamento, Triagem e Curação

Para maximizar a eficiência financeira e operacional da infraestrutura de IA da Koder Stack, a métrica normativa KVIQ (definida em specs/ai/kviq.kmd) é integrada aos componentes do sistema de curadoria e roteamento sob três formas de uso:

1. Roteamento Dinâmico no Gateway (services/ai/gateway):

   O gateway de IA passa a resolver codinomes e apelidos dinâmicos baseando-se em cabeçalhos de requisição (como X-Koder-Strategy) ou configurações do tenant no SDK do resolver (modelreg#005): * max-intelligence: Roteia para o modelo de maior score absoluto no SWE-bench Pro (ex: claude-4.8-opus). * max-efficiency: Roteia para o modelo com maior KVIQ Score ativo (ex: deepseek-v4-pro ou gemini-3.5-flash). * hybrid-context (Context-Aware): Se a contagem de tokens do prompt exceder \(150K\) tokens (indicação de ingestão massiva de monorepo), roteia automaticamente para o modelo de grande contexto com melhor KVIQ (ex: gemini-3.1-pro); caso contrário, usa a rota padrão de alta eficiência.

2. Triagem de Evals no Curator e Eval (services/ai/eval - Loop B):

   Para conter o uso computacional de GPUs de testes locais (s.khost1), o sistema de avaliação executa um pré-filtro (KVIQ Triage Gate) antes de admitir novos modelos descobertos em fontes externas (Loop B) para a suíte completa: * Computa-se um Triage-KVIQ preliminar usando metadados públicos do modelo (janela de contexto, preço anunciado e aproximação de score lógico via ELO/MMLU-Pro). * Apenas modelos que alcancem um Triage-KVIQ \(\ge 35\) são promovidos para a suíte de testes de código reais (full tier). Modelos abaixo deste limite são marcados em modelreg apenas como catalogados, poupando ciclos de GPU.

3. Gatilho de Depreciação Automática no Curator (Loop A):

   O KVIQ serve como métrica de obsolescência relativa. Se o KVIQ de um modelo ativo cair abaixo de \(25\%\) do KVIQ do modelo líder da sua respectiva categoria, o curator dispara automaticamente uma proposta de desabilitação e demissão do modelo (DeprecationProposal no desk).

9. Edits to normative artifacts required to ratify

1. registries/component-names.md + ticket-prefixes.md — register the new sector.
2. New registries/ai-benchmarks.md (+ schema) — benchmark relevance source of truth.
3. modelreg/types.go — FamilyMinimax; video modalities; PromotionProposal.
4. registries/ai-model-recommendations.md — note the curator as a writer

   (alias picks become curator-proposed, human-approved).

5. meta/docs/stack/specs/ai/kviq.kmd — formalize KVIQ calculation and bands (Landed, v1.0.0).
6. meta/docs/stack/vocabulary.md — register KVIQ term in central vocabulary (Landed).
7. On loop maturity: an amendment enabling guardrailed auto-promotion (D1).