AI Gateway

• Area: Intelligence
• Path: ai/gateway
• Kind: Unified LLM proxy / router (OpenAI-compatible API)

Role in the stack

AI Gateway is the single entry point for any Koder product that needs to call an LLM. It exposes an OpenAI-compatible API and routes requests to the right backend — local LLMs (via ai/runtime + Ollama), or proprietary providers (OpenAI, Anthropic, Google Gemini, HuggingFace, vLLM, Stability, Deepgram) when enabled in config.

Running in production on s.r1 as LXC 128, internal endpoint http://10.0.1.128:7800/v1/. Consumers today include ai/voice (for transcription summarization); new products plug in by pointing their SDK at the gateway without hardcoding provider keys.

Parallel task dispatch (AIGW-051 governor + AIGW-052 HTTP/SSE edge)

internal/dispatch is the multi-tenant concurrency governor for parallel agent sub-runs (AICORE-119 Manus-parity axis): it admits sub-runs under per-user + per-org budgets (LimitsForTier: free 1 / pro 5 / team 20 per user, aligned with ai/runtime's orchestrator), FIFO-queues the excess, fans the artifacts in, and merges each child's event stream annotated by ChildID. The per-user cap stays below the org ceiling so one user never monopolises an org's slots (tenant A never blocks tenant B). The Executor is an interface — the real one forwards sub-runs to ai/runtime's RUNTIME-031 orchestrator.

internal/dispatchapi (AIGW-052 API edge — DONE 2026-05-25) is the HTTPSSE surface over the governor: `POST /apiv1agentparallel accepts {"parallel":[{prompt,tools},…]}, resolves the caller's tier → budget, assigns child ids, and streams event: child per sub-run event (carrying child_id/seq/kind) then a final event: done with the aggregated fan-in; over-budget callers get **429 + Retry-After** (pre-admission load-shed against Stats()); GET apiv1agentparallelstats exposes the caller's live active counts for the "N in queue" UI. Tenant resolution is injected (TenantResolver, default reads auth-middleware context) so the edge is unit-tested with a fake executor (6 tests, -race). **Remaining integration half (AIGW-052):** the real runtime-forwarding Executor, route registration in cmdgateway, and live SSE e2e + KAiParallelTabs` UI.

Scheduled tasks (AIGW-050)

internal/schedule is the scheduled-agent-tasks engine (Manus "Agendado" sidebar): a dependency-free POSIX cron parser (5-field + @daily@hourly@weekly… macros, listsranges/steps, Vixie dom-vs-dow rule), the agent_schedules model + tenant-scoped store, per-tier caps (MaxSchedulesForTier: free 0 / pro 20 / team ∞), and a clock-injectable Scheduler with Tick (fire due + advance) and CatchUp (a miss within 24h fires once; older is skipped).

internal/scheduleapi (AIGW-053 schedule CRUD slice — DONE 2026-05-25) is the HTTP surface over that engine: POST/GET/DELETE/PATCH /api/v1/agent/schedules[/{id}] for createlistgetdeleteenable, tenant-scoped + tier-enforced (Store.Create parses the cron, computes NextRun, enforces the cap; the handler maps invalid-cron → 400, free-tier → 403, over-cap → 403, cross-tenant → 404 per the multi-tenancy contract). Tenant+tier come from the auth context (never the body); 7 tests under -race.

internal/trigger (AIGW-053 inbound-trigger framework + webhook-signed adapter — DONE 2026-05-25) is the inbound side: a small framework (EndpointResolver + Dispatcher interfaces, buildPrompt) plus the signed-webhook adapter. POST /api/v1/triggers/webhook/{endpoint} resolves the endpoint (unknown → 404), caps the body (1 MiB → 413), verifies a constant-time HMAC-SHA256 signature (X-Koder-Signature: sha256=<hex>, 401 on mismatchmissingtampered), then dispatches an agent run as the endpoint's tenant (202; dispatch failure → 502). VerifySignatureSign are the symmetric primitives; 9 tests under -race. The github-PR-comment adapter (github.go) reuses the framework: `POST /apiv1triggersgithub{endpoint} verifies GitHub's X-Hub-Signature-256 HMAC, 200-acknowledges ping`non-commentnon-createdkeyword-less events without dispatch (no retry), and on a /koder-keyword created comment strips the keyword, prefixes the repo#PR context, and dispatches as the tenant (202); +7 tests. The slack-mention adapter (slack.go) verifies the Slack v0 request signature (X-Slack-Signature: v0=<hex> over v0:{ts}:{body}) with a ±5-min anti-replay window (X-Slack-Request-Timestamp, clock injectable), echoes the url_verification challenge, ignores non-app_mention events with 200, and on an app_mention strips the <@…> mention(s) + prefixes the channel and dispatches as the tenant (202); VerifySlackSignatureSlackSign primitives, +7 tests. The trigger suite is 23 tests under -race. The *ersistent schedule.Store (`internalschedulepostgres.go, AIGW-053 — DONE 2026-05-28, /k-go drain)** is a PGStore over pgxpool mirroring MemStore field-for-field (cron-parse + tier-cap in Create inside a tx, multi-tenant scoping → cross-tenant ErrNotFound, Due system scan, MarkRan advance); RunMigrations uses a dedicated gatewayschedulemigrations table so it never collides with internalusage's migrations on a shared DB; integration test /o:build integration (GATEWAYSCHEDULETEST_DSN). **Remaining (AIGW-053, runtime/serve-time):** the real EndpointResolver (endpoint registry + servicesfoundationid linked accounts) + real Dispatcher (AIGW-051/RUNTIME-031), the 2 infra-coupled adapters (email/jira), the per-minute Scheduler.Tick/CatchUp loop, and cmd/gateway` route wiring.

Primary couplings

Providers (configurable)

• Enabled by default: local Koder AI Runtime (Ollama) on LXC 127
• Available when enabled: OpenAI, Anthropic, Google Gemini, HuggingFace, vLLM, Stability, Deepgram

Interfaces

• OpenAI-compat /v1/ API (chat, completions, embeddings)
• YAML configuration at /etc/koder-ai-gateway/config.yaml
• systemd service koder-ai-gateway
• Run notifications (AICORE-144 + observe-RFC-002 / notify#010) — the gateway

  POSTs run-lifecycle pushes to self-hosted koder-notify (/api/v1/notify, no FCM) and proxies the WebPush subscription lifecycle (POST /v1/webpush/{subscribe, unsubscribe} → koder-notify, JWT-scoped to the caller) so the fan-out has subscriptions to reach. The legacy local FCM-style device registry (internal/devices) is retained pending retirement per RFC-002 §8.

Storage

Caminho ratificado em backlog/done/047-tut-discovery-sqlite-vs-postgres.md (caminho A — migrar pra Postgres antes do tracker centralizado RFC-001). Migration plan em backlog/pending/048-.... Detalhe de current state em docs/technical/storage.kmd.

Agent stream protocol (2026-05-16)

The unified Koder AI product (services/ai/ai/) ships its own gateway that proxies the OpenAI-compatible API AND exposes the Koder Agent Stream Protocol for live agent run subscription.

• Route: GET /v1/agent/runs/{id}/stream — auto-detects SSE vs WebSocket upgrade
• Wire: 7 event types (reasoning.delta, text.delta, tool.start, tool.end, step.boundary, child.spawn, run.lifecycle)
• Throughput shaping (R1-R4): 100ms aggregation by default; ?detail=full opts out
• Reconnect: Last-Event-ID (SSE) / {op:"resume",seq:N} (WS) replay from persistent store
• Auth + tenancy: JWT required (API-key auth refused for streams); cross-tenant → 404 per specs/multi-tenancy/contract.kmd
• Persistence: RunEventStore interface with two backends — InMemoryRunEventStore (default, boot-safe, ephemeral) or KDBRunEventStore (opt-in via RUN_EVENT_STORE=kdb + KDB_BASE_URLKDB_API_KEYKDB_NAMESPACE). KDB backend runs idempotent CREATE TABLE/INDEX IF NOT EXISTS from migrations/001_agent_run_events.sql on boot. Cross-tenant reads return 0 rows → handler surfaces 404 via CountForTenant. Retention: 90d done / 30d aborted, pruned hourly.
• Reference impls:
  • Go: services/ai/ai/gateway/internal/stream/{events,hub,sse,ws,aggregator,store,batchwriter}.go
  • JS: engines/sdk/js/agent-stream/index.js (exported as @koder/sdk/agent-stream)
• Tests: 29 Go sub-tests + 7 JS conformance tests, all green under -race
• Tickets: AICORE-120 (parent) → AICORE-127, 128, 129, 130, 131, 132 (all closed 2026-05-16)

Run control plane (2026-05-16)

Companion to the agent stream protocol: REST surface for the out-of-band controls Manus / Devin 2.0 / Cline expose (pause, redirect, rewind, planning gate, follow-up suggestions).

• State machine: see services/ai/ai/gateway/internal/runctl/types.go — 8 states + transition table; terminal states reject all transitions.
• Planning gate modes: off (Claude-Code-classic), preview (default; agent emits plan and starts running), required (blocks on approval). The runtime decides which based on workspace settings + request.
• Multi-tenancy: every endpoint enforces JWT-bound tenant via auth.ClaimsFromContext; cross-tenant resolution returns 404 per specs/multi-tenancy/contract.kmd.
• Error codes: AICORE-CTL-0001..0008 and AICORE-CTL-0099 per specs/errors/user-facing-messages.kmd.
• Persistence: runctl.Store interface (InMemory ships today); kdb-backed impl tracked alongside AICORE-133.
• Spec extension: agent-stream.kmd v0.2.0 adds the plan.proposal event type.
• Tickets: AICORE-117, 121, 122, 123, 126 (gateway slice closed 2026-05-16; runtime consumer slice AICORE-134134.1134.3 closed 2026-05-18; checkpoint+followup concrete impls 134.2134.4 + KITWEBKIT widgets 134.5/134.6 still pending).
• Redirect intent persistence (AICORE-134.1): /redirect and /rewind routes now persist a runctl.RedirectIntent (message + resolved rewind target) before transitioning state. The runtime fetches+clears the intent in waitForRunnable and jumps the Execute loop index when a checkpoint resolved.

Agent runtime (2026-05-16, AICORE-137 first slice)

The muscle that consumes everything shipped above. internal/agent/ owns a Runtime that drives a Plan of (tool, input) steps end-to-end and surfaces the protocol events.

• Tool interface: Name() / Description() / Call(ctx, input) (output, error). Implementations MUST be concurrent-safe. Built-in EchoTool for smoke tests.
• ToolRegistry: concurrent-safe map; Register / Get / Names / Len. Overwriting is allowed so tests can swap impls.
• Runtime.Execute(ctx, Plan):
  • State pre-check: rejects unless run is in Running / Paused / Redirecting.
  • For each step: waitForRunnable (polls runctl.Store every 250ms while Paused) → emit step.boundary → emit tool.start → Tool.Call(ctx, input) (60s timeout) → emit tool.end (ok + duration_ms, or error with KAI-TOOL-CALL-0001).
  • Redirecting consumes the persisted runctl.RedirectIntent (set by /redirect or /rewind), clears it, transitions back to Running, and when the intent carries a resolved RewindToCheckpointID jumps the loop index to the matching step (AICORE-134.1).
  • Aborted during pause returns ErrAborted; tool errors transition the run to StateError; unknown tools emit tool.end with KAI-TOOL-LOOKUP-0001 and return ErrToolNotFound.
  • Clean completion transitions to StateDone and emits final run.lifecycle done.
  • Checkpoints (AICORE-134.2): Runtime.Checkpointer is a pluggable CheckpointCapture invoked between successful steps; default nil (no checkpoints), NopCheckpointCapture{} enables timeline-only.
  • Follow-ups (AICORE-134.4): Runtime.FollowUps is a pluggable FollowUpGenerator invoked after Done to populate /follow_ups; default nil (no suggestions). Validates via runctl.ValidateFollowUps (3–5 items).
  • PlanAndExecute (AICORE-134.3): alternate entry that wires Planner → runctl.Plan save → PlanningGate flow → Execute. IntentFromTool keyword-classifies steps before SavePlan so UIs render the right icon.
• Sink interface mirrors *stream.PersistentSink.Publish so the runtime is decoupled from the Hub + persistence types. Production binds the gateway singleton; tests inject captureSink.
• Tests: 10 sub-tests verde under -race (ToolRegistry, EchoTool, Execute happy path / lookup error / paused-then-resumed / aborted / redirect / bad initial state / missing run_id / tool error).

The runtime deliberately leaves the LLM-driven planner to AICORE-137b — this slice ships the executable primitive; planning ships on top of it.

Default tool registry (AICORE-137d)

NewServer builds the agent runtime with agent.DefaultRegistry(fetcher, pipeline), exposing three tools out of the box:

Tool inputs mirror the REST body shapes of /v1/tools/web_fetch_rich and /v1/tools/web_extract so Plans can reuse payload templates verbatim. Server.AgentRuntime() and Server.AgentTools() are public accessors for the admin handlers below.

Agent admin endpoints (AICORE-137e)

• Body: {steps: [{tool, input?, id?}], prompt?, planner_options?, planning_gate?: off|preview|required, workspace_id?}. Steps OR Prompt must be set (Steps wins when both — debug path). Missing tool names or unknown tools return 400 with KAI-AGENT-PROTO-0003/0004 before any run is created.
• Async semantics: 202 returns immediately; the runtime runs on a goroutine with a 1h ceiling. Clients subscribe to stream_url to observe progress.
• Planning gate: offpreview start the runtime immediately; required parks the run in awaiting_approval, auto-saves a passthrough runctl.Plan, and a background watcher polls (10min max) for the user's `POST ...approve_plan before firing Runtime.Execute`.
• Tenancy: every run is registered with stream.RunRegistry for the authenticated koder_user_id so the live stream route enforces cross-tenant → 404.

This is the end-to-end user surface for the agent — the cycle from "I want X done" → planning → execution → live event stream → completion now closes inside the gateway.

LLM planner (AICORE-137b)

When POST /v1/agent/runs body carries prompt instead of steps, the gateway asks the LLM to turn the prompt into an executable Plan. The planner reads the same tool catalog /v1/agent/tools lists; every step is validated against the registry before any run is created (defense against tool hallucination + prompt-injection of unplanned capabilities).

• Default model: claude-haiku-4-5-20251001 (cheap + JSON-tight). Override per-call via planner_options.model or globally via EXTRACTOR_MODEL.
• Step cap: planner_options.max_steps (default 8). LLM replies are truncated.
• JSON tolerance: the parser strips json fences and accepts both bare arrays [...] and wrapper objects {steps: [...]} / {plan: [...]} / {actions: [...]}. Accepts both tool and name field aliases. Step IDs auto-assigned (s1, s2`, ...) when LLM omits.
• Error codes: KAI-AGENT-PLAN-0001 (planner not configured) / KAI-AGENT-PLAN-0002 (planner returned error: empty prompt, unknown tool, malformed reply, chat upstream failure).
• Tests: 15 sub-tests in internal/agent/planner_test.go + 3 route-level tests covering prompt-driven happy path, no-prompt-no-steps 400, planner-error 502.

Production binds LLMPlanner to the same LoopbackChatCompleter that backs the LLMExtractor (AICORE-135) — single egress path, single retry policy, same observable surface.

Self-repair loop (AICORE-116)

When a tool call fails AND the configured Matcher(toolName, err) returns true, the runtime re-prompts the planner with the failure context and runs the replacement step. Default-on for test-runner tools (pytestjestplaywrightgotestflutter_test/etc.) with MaxIterations=3.

• Config struct: agent.RepairConfig{Planner, Matcher, MaxIterations}. Zero-value disables repair (back-compat with #137).
• Default: agent.TestRunnerMatcher() — matches any tool name containing one of 11 test-runner tokens. Extend via TestRunnerMatcher("custom_runner").
• Replacement step IDs: <original>_repair_<N> so emitted step.boundary / tool.start / tool.end events don't collide on the wire.
• Telemetry: RepairTelemetry.Snapshot() returns (attempts, repaired, failed) atomic counters. Prometheus exposed via koder_ai_test_repair_attempts_total{outcome} (AICORE-116c) + koder_ai_test_repair_iterations{outcome} histogram (AICORE-116e).
• Audit log (AICORE-116d, 116f): every attempt + final outcome emits one agent.repair slog Info entry with {run_id, koder_user_id, tool, step_id, attempt, outcome, original_error, iterations, ts}. REPAIR_AUDIT_BACKEND=kdb opts into agent.KDBAuditSink (persistent agent_repair_attempts table + 12-month PruneOlderThan retention). Misconfig falls back to slog with WARN. Write path is fire-and-forget so audit infra outages don't disturb the runtime.
• Override: Server.AgentRuntime().Repair = ... before serving traffic to tighten/widen the matcher or disable repair entirely.

Streaming tool output (AICORE-137c)

Tools that produce text-shaped output (markdown, generated content, LLM-extracted streams) can opt into incremental delivery by implementing the StreamingTool interface:

type StreamingTool interface {
    Tool
    CallStream(ctx, input, chunks chan<- string) (json.RawMessage, error)
}

When the runtime encounters a StreamingTool, every value sent on chunks becomes a text.delta envelope between tool.start and tool.end. The final JSON return value is still surfaced via tool.end.output, so replay + persistence remain canonical.

• Backward compatible: non-streaming tools (only Tool) run unchanged — no text.delta events emitted.
• Reference impl: StreamingEchoTool (registered as echo_stream) splits input on whitespace, emits one chunk per word; useful for UI smoke-tests of streaming consumers.
• Cancellation: ctx.Done() aborts streaming cleanly mid-flight.

Provider plane (2026-05-16, AICORE-118)

Provider interface gained two optional extensions and the registry grew a factory layer:

• CatalogProvider — Models(ctx) ([]ModelInfo, error). Implemented by all 6 providers (OpenAI, Anthropic, Gemini, Moonshot, Manus, OpenRouter) via static catalogs in providers/catalog.go.
• HealthChecker — HealthCheck(ctx) error. HEAD against each provider's models endpoint; auth-gated 401/403 counts as healthy.
• Factory registry — Registry.RegisterFactory(name, func(apiKey) Provider) + BuildFromKeys(map[name]apiKey) replaces the long if cfg.X != "" { Register(NewX(cfg.X)) } chain in main.go.
• Registry.AggregateModels(ctx) — fans out to every CatalogProvider; 1h cache; sorted by (OwnedBy, ID). Backs GET /v1/models (previously a hardcoded list).
• Registry.Health(ctx) — fans out to every HealthChecker; non-implementers reported as OKfalse + `error"not implemented" (vs error=<probe failure>`).
• New route: GET /v1/providers/health — JSON aggregate.

Web tools (2026-05-16, AICORE-113, AICORE-114)

Two LLM-tool endpoints on the gateway, std-lib only (no chromedp / no readability dep yet).

• Body cap: 5MB on web_fetch_rich (MaxBodyBytes).
• JS render: opts.js_render=true returns 501 KAI-WEB-FETCH-JS-0001 until chromedp engine ships (AICORE-135).
• Schema dialect (MVP): {name: "string", price: "number", in_stock: "boolean"} — full JSON Schema in follow-up. Unsupported types return 400 KAI-WEB-EXTRACT-SCHEMA-0001.
• LLM step (AICORE-135): LLMExtractor wires to a ChatCompleter interface; production binds LoopbackChatCompleter (POSTs to the gateway's own /v1/chat/completions — reuses provider routing, retries, auth). Default model claude-haiku-4-5-20251001 overridable via EXTRACTOR_MODEL. Loopback URL/key via EXTRACTOR_BASE_URL / EXTRACTOR_API_KEY (defaults: in-process URL + any registered kgw_* key).
• Caching: web_extract keys on sha256(url + sorted schema), default TTL 1h (opts.cache_ttl_secs < 0 disables, 0 = default).
• Auth: JWT-only (API keys forbidden — outbound HTTP is per-user-rate-limited). All routes use auth.ClaimsFromContext.
• Error codes: KAI-WEB-AUTH-0001, KAI-WEB-FETCH-0001/0002/0099, KAI-WEB-FETCH-JS-0001, KAI-WEB-EXTRACT-0001/0002/0099, KAI-WEB-EXTRACT-SCHEMA-0001.
• Tickets: AICORE-113 (rich fetch), AICORE-114 (extract), AICORE-118 (provider refactor backing both), AICORE-135 (LLMExtractor wireup) — all closed 2026-05-16. Follow-up AICORE-136 (chromedp JS render).

Checkpoint retention + erasure cascade (2026-05-23, AICORE-134.2c + AICORE-138)

The retention surface that backs the agent runtime's CheckpointCapture pipeline. Policy lives in meta/docs/stack/policies/checkpoint-retention.kmd; this section is the operator's view.

• Per-run cap (R1): agent.EnforcePerRunCap runs inline in maybeCheckpoint after every successful capture. Default 10 envelopes per run; RETENTION_PER_RUN_CAP=-1 disables (debug only).
• Per-tenant cap (R2): agent.EnforcePerTenantCap sweeps oldest-across-runs until the tenant's total SizeBytes ≤ cap. Default 500 MiB (RETENTION_PER_TENANT_BYTES).
• Done-run grace (R3): agent.EnforceGraceExpiry drops every checkpoint belonging to terminal runs older than the configured window. Default 7 days, clamps at 90 (RETENTION_DONE_RUN_GRACE_SECS).
• Periodic worker: Server.StartRetentionTicker spawns the cross-tenant ticker that runs R2 + R3 every RETENTION_INTERVAL (default 15 min). TenantSource derives from InMemoryRunRegistry.Tenants() — kdb-backed registries will plug in via the same tenantLister interface.
• Multi-tenant isolation (R4): every RetentionStore call passes koder_user_id; cross-tenant deletes mask to silent nil per multi-tenant-by-default.kmd § 5.
• Erasure cascade (R5, AICORE-138): gateway/internal/erasure/Subscriber consumes identity.user.erased envelopes from the koder:events:id Redis Stream (cross-service-events.kmd canonical type + stream key). On every erasure event, agent.EnforceErasure walks every checkpoint owned by the subject koder_user_id and drives deleteOne with ReasonErasure. Idempotent — re-delivery on an emptied tenant is a noop. Wired in main.go when REDIS_URL is set; empty REDIS_URL keeps R1R2R3 local enforcement without R5. Consumer group: koder-ai-gateway-erasure (pattern <service>-erasure-cascade per cross-service-events.kmd § R3).
• Audit log (R6): every deletion emits a structured slog Info entry event=checkpoint.deleted run_id=… checkpoint_id=… koder_user_id=… size_bytes=… reason=… where reason ∈ {per_run_cap, per_tenant_cap, grace_expired, erasure}. Shared deleteOne helper guarantees the same shape regardless of which enforcer fired.
• Blob store: agent.FsBlobStore rooted at CHECKPOINT_DIR (default /var/lib/koder-ai-gateway/checkpoints); content-addressed (<tenant>/<scheme>/<sha256>); cross-tenant Get/Delete mask to ErrBlobNotFound.
• Restore path: Server.SetRewindRestorers plugs a RestorerRouter; default boot wires NopRewindRestorer + TarZstdRewindRestorer. OverlayfsRewindRestorer (AICORE-134.2b) blocked on AICORE-106 sandbox-model RFC.
• Tickets: AICORE-134.2c (BlobStore + RewindRestorer + R1R2R3R4R6 + periodic worker + main.go wiring) closed 2026-05-19; AICORE-138 (R5 erasure cascade subscriber) closed 2026-05-23. AICORE-134.2b (overlayfs producer + restorer) still pending.

Per-identity provider-key resolution (2026-05-31, AIGW#056)

The gateway resolves a caller's stored BYO provider key (OpenAIGeminiAnthropic/…) by identity and scope precedence, per stack-RFC-007 §6. It is the gateway-side consumer of the Koder ID keys service's gRPC ProviderKeysService.ResolveProviderKey (the server-to-server-only surface that returns the plaintext secret; the HTTP surface is user-CRUD only).

• Key precedence in provider.effectiveKey: per-request BYOK header → resolved per-identity key (user → workspace → org, applied server-side by id#194) → static Stack-managed key. The resolution tier is installed via a process-wide provider.SetKeyResolver hook (reuse-first: no per-factory signature churn) and read by openaianthropicgoogle effectiveKey (covers the OpenAI-compatible siblings — DeepSeekxAIMistralKimiGroqPerplexityTogether).
• Resolver (internal/providerkeys): gRPC client + short-TTL cache (default 60s; positive and negative caching so the common no-BYO-key path doesn't hammer the service) + scope metric. Fail-open: a genuine miss (NotFound — also covers cross-tenant, which the keys service masks to NotFound, never leaking existence) falls through to the static key; a transient keys-service outage does not hard-fail the request (it's not cached, retried next request). Rotation reflects within CacheTTL; Invalidate available for explicit signals.
• Identity carrier: the request handler injects provider.Identity{UserID, WorkspaceID (X-Workspace-ID), OrgID} into the request context next to BYOK injection; the resolver reads it.
• Config: [keysService] enabled + address + cacheTTL + timeout; the startup dial is optional and fail-open (disabled config or dial failure = prior static-key behavior). go.mod consumes the id engine pkg module via replace (the "pkg is the Go SDK" pattern) for the generated keysv1 client.
• Metrics: aigw_provider_key_resolution_total{scope} (scope ∈ user|workspace|org|none) + aigw_provider_key_resolution_unavailable_total.
• Tickets: AIGW#056 closed 2026-05-31 (closes id#194's gateway acceptance); follow-up AIGW#059 (billing attribution to the resolved key's owner — the resolver already reports the scope; usage-tracker wiring pending).

Workspace-scoped API keys + RFC-017 §7 quota cascade (2026-06-02, AIGW#058)

The gateway is the request-time enforcement consumer for the RFC-017 §7 tenancy-limits cascade; the authoritative resolver lives in the id engine (engine#199). The gateway-side machinery:

• internal/tenancy (new): MinLimit(org, workspace) is a byte-for-byte replica of the id engine's canonical pure cascade (min(org, workspace), NULL = unlimited, workspace may only tighten). It is duplicated — not imported — because the canonical copy sits in the id engine's internal/ service package, unimportable across module+internal boundaries; the ticket sanctions this precisely because the function is pure. Plus OverLimit (inclusive used ≥ limit), the Resolver seam (EffectiveLimitResolveWorkspace) with an Unlimited default, a UsageReader seam, and QuotaEnforcer which fails open on any resolverusage error (always-on posture — a tenancy-service blip never takes the data plane down).
• Key scope (internal/middleware/apikeys.go): the canonical AI api_keys store gains workspace_idproject_id, carried through CreateValidateListRotate. Migration is additive + legacy-safe (duplicate-column-tolerant ALTER TABLE) so existing apikeys.db files upgrade in place. POST /api/v1/keys accepts the two new fields.
• Default-workspace absorption: CombinedAuth.withWorkspace resolves a legacy org-only key onto the org's default workspace at read time, without mutating the key row (RFC-017 §7), on both managed-key and OIDC paths; the effective workspace lands on UserInfo.WorkspaceID + a workspaceIDKey context value (GetWorkspaceID). Default tenancy.Unlimited resolver = explicit workspace passes through, org-only stays personal — exactly the pre-#058 behaviour until the id-backed resolver is wired via CombinedAuth.SetResolver.
• Quota enforcement: the tiered rate limiter gains an optional QuotaEnforcer (SetQuotaEnforcer); admission enforces the min(org, workspace) monthly ceiling → 429 monthly_token_limit_exceeded. nil enforcer = no monthly-ceiling enforcement.
• Serve-time seam (remaining): the real Resolver over the id limits endpoint (idengine#200) + real UsageReader (month-to-date from `internalusage), wired in cmdgateway. Until then the gateway runs Unlimited (quota enforcement is a no-op, backward compatible). The store now carries workspace_id`, so idengine#200's "archive workspace revokes its keys" lockstep can scope a future revoke-by-workspace path.
• Tickets: AIGW#058 closed 2026-06-02; built + race-tested on dev-linux-box.

Billing attribution to the resolved key owner (2026-06-02, AIGW#059)

Closes the loop opened by AIGW#056: when a workspace/org BYO provider key pays for a request, the spend is credited to that owner — not only the triggering user (stack-RFC-007 §6).

• Attribution sink (internal/provider): Attribution{Scope} + WithAttributionAttributionFromRecordResolvedScope — a per-request pointer the request handler installs before dispatch and the key resolver (deep in provider dispatch) fills with the scope of the key it resolved. The httptrace pattern: a downstream callee records into the same struct the upstream handler reads, with no hot-path signature churn.
• Resolver (internal/providerkeys): on a positive resolution records the resolved scope (userworkspaceorg) into the sink — on both the fresh RPC and the cache hit (the cache entry now carries the scope, so the common 60s-cached path still attributes correctly). A miss leaves the sink empty.
• Usage event (internal/usage): Event gains WorkspaceID (RFC-017 §6 materialized path) + KeyOwnerScope; persisted in the usage_events log on sqlite (additive ALTER) and postgres (migration 0004_usage_event_attribution).
• Handler classification (cmd/gateway): on Record sets WorkspaceID (the #058-absorbed workspace, X-Workspace-ID fallback) and KeyOwnerScope = byok (per-request header) | the resolved per-identity scope | stack (Stack-managed key).
• Deferred: re-graining the usage_dailyusage_monthly rollups + the billing UsageRollupQuery (RFC-017 §7 SUM) to key on workspace_id — that aggregation lives in `servicesfoundation/billing`; the event log now carries the axis. Quota consuming the resolved owner is the #058 serve-time seam (id#200).
• Tickets: AIGW#059 closed 2026-06-02; also fixed a pre-existing test-mock data race in internal/usage/ratelimit_test.go.

Grounded-search capability (2026-06-02, AIGW#057)

A dedicated route (grounding is not a chat-completion) that returns source citations from a grounding-capable provider, per stack-RFC-007 §7. The first consumer is edictus' aioverview; the requestresponse shapes were ported from that field-tested code (it lives in another module's `internal`, so unimportable).

• POST /v1/grounded-search {query, max} → {citations:[{title, url, confirmed:false}]}. Citations come from the provider's groundingMetadata.groundingChunks[].web, never the model prose, and are always confirmed:false — the consumer must validate each URL against its source.
• internal/groundedsearch: calls Gemini generateContent with the google_search tool, dedups, caps at max (default 10), and resolves vertexaisearch …/grounding-api-redirect/… URLs to their final source (best-effort, 5s-bounded, falls back to the raw URL).
• Key resolution: no provider key in the payload — the handler resolves the google key by identity via the shared provider.EffectiveKey(ctx, "google", static) (BYOK header → per-identity AIGW#056 → static Stack key). That helper was extracted from the per-provider effectiveKey methods (googleopenaianthropic now delegate to it — DRY).
• Metrics: aigw_grounded_search_total{status} + aigw_grounded_search_citations_total; latency on the existing duration histogram (provider=google,model=grounded-search).
• Provider fallback (when the primary lacks grounding) is a future seam — google is the only grounding provider today.
• Tickets: AIGW#057 closed 2026-06-02.

Agent-run persistence + skill-mining read API (2026-06-02, AIGW#055)

The gateway owns the agent-run record but had no queryable store; this adds one so the skill-mining worker (tools#020) can read recurring prompt+tool-sequence patterns.

• internal/runs (new): Run{ID, KoderUserID, OrgID, WorkspaceID, Prompt, Tools[], Rating, At} + a Store (RecordRateTenantsTenantRuns) and a write-only Recorder seam. SQLite-backed (agent_runs table, file-backed — the gateway's sqlite-first posture, same as apikeys); a Postgreskdb impl can drop in behind Store for scale. Rate is tenant-scoped → ErrNotFound on cross-tenant/absent (multi-tenant-by-default §5, no existence leak).
• Read API (service-account gated, mirroring the smart-router admin routes): GET /v1/runs/tenants and GET /v1/runs?tenant=&since= return the exact mining.Run JSON projection (id, koder_user_id, prompt, tools, rating, at) the worker's RunSource decodes. The worker iterates all tenants → a privileged service read.
• Rating endpoint: POST /v1/runs/{id}/rating {rating:1..5} — caller-authenticated, tenant = the authenticated user, cross-tenant/absent → 404 (the AICORE-124 rating feed).
• Recording seam: dispatchapi.Handler.WithRecorder persists each completed parallel sub-run (prompt + ordered tools + tenant, unrated); nil = no-op. End-to-end recording activates when the parallel-dispatch routes are wired in prod (the serve-time half of AIGW#052) — the store + read API are the standalone prerequisite and are live now.
• Config: runs.databasePath (default runs.db).
• Follow-up (different component): the worker's HTTP RunSource impl in services/ai/tools (replacing FakeRunSource) + the 3-rated-runs end-to-end mining test.
• Tickets: AIGW#055 closed 2026-06-02.