AI Runtime — Local LLM Serving

• Area: Intelligence
• Path: ai/runtime
• Kind: LLM runtime (download, serve, quantize, fine-tune)

Role in the stack

AI Runtime is the local LLM serving layer of Koder. It downloads models from ai/zoo (or upstream providers), serves them via an OpenAI-compatible API, quantizes them for the target hardware, and optionally fine-tunes them on tenant data. Its consumers are ai/gateway (which routes traffic to Runtime as one of its backends) and any service that needs local inference without leaving the Koder perimeter.

Deployed in production as LXC 127 on s.r1 using Ollama as the inference backend.

Features

• Download + serve models from ai/zoo or upstream registries
• Quantization for GPU / CPU targets
• Fine-tuning pipeline
• OpenAI-compat /v1/ API
• Scales from laptop to cluster
• Parallel agent-execution tier (internal/orchestrator, RUNTIME-031) — fans a task into N concurrent sub-runs across a hot-reloadable worker pool, streams each child's events back annotated by ChildID (per-child ordered), fans the artifacts in, applies per-child timeouts, signals backpressure, and exposes queue_depthworker_idlerun_duration_p95. Queue + Runner are interfaces (in-memorystub now; Redis Streams + sandbox-backed runner in RUNTIME-032). This is the foundation AIGW-051 (`aigateway`) exposes as the parallel-dispatch API — part of the AICORE-119 Manus-parity umbrella.
• Model catalog is curator-owned (RUNTIME-033, stack-RFC-008): the runtime is a consumer of models.yaml, not the home of discovery. services/ai/curator performs governed, multimodal discovery and writes approved enabledisableversion entries back (curator#008's comment-preserving writer, inline # curator: provenance). The runtime's internal/updater keeps only the weekly SHA-freshness loop — it reads the catalog (honouring enabled + consuming version) and persists SHA to a separate updater-state.json, so the two writers never collide. The old text-only DiscoverNew / --discover is retired (returns ErrDiscoveryRetired + a deprecation notice pointing to curator). Standalone-safe: curator-unavailable just keeps the current models.yaml (modelreg#005 fallback).

Primary couplings