stack-RFC-020 — AI compute strategy: rent the kernels, own the layers, shadow the vendor-neutral runtime

accepted

Makes explicit the Koder Stack's long-term AI model-COMPUTE strategy (the tensor kernels for inference + fine-tuning), which is already implicit in its existing components + the self-hosted-first shadow-flip model. Decision: (a) the mature external GPU ML stack (CUDA/ROCm + axolotl/unsloth/transformers for fine-tuning; Ollama/llama.cpp/vLLM for inference; frontier APIs for the hardest reasoning) is CANONICAL, because it is the only option that passes self-hosted-first G2 (perf parity) today; (b) a Koder-owned, vendor-neutral Rust-ML runtime (candle/burn over wgpu+CUDA) is run as a tracked SHADOW-ACTIVE maturation track that flips per case-of-use only when it clears G2 (≤×1.05) + G3 (≥3 stable releases) — exactly the kodec model. The Stack OWNS the layers where it is already competitive (serving/orchestration/data: gateway, kdb-vector, rag, memory — native Rust/Go). This is ORTHOGONAL to Kanvas (GPU-for-UI ≠ GPU-for-tensors).

Companion to the conversation that produced stack-RFC-019. The owner asked whether the Stack's GPU+native direction speeds up AI. Answer: it speeds up the infra AROUND the model (real), not the model math (which needs a dedicated GPU compute stack). This RFC makes the compute strategy explicit.

1. The distinction that frames everything

GPU-for-UI (Kanvas/wgpu graphics) ≠ GPU-for-AI (tensor compute). They are different uses of the same hardware. Kanvas does not accelerate models. And "native compilation" (Rust/Go host) accelerates CPU-side glue, not the tensor math — that runs in GPU kernels (CUDAcuDNNetc.), independent of the host language.

2. Where the Stack already is (this RFC formalizes, not invents)

  • Inference: the AI gateway routes via aliases to frontier APIs

    (ClaudeGPT) + *ocal Ollamallama.cpp*(GGUF on CPU/GPU). Self-hosted at the serving level; the kernels are llama.cpp/CUDA.

  • Training: services/ai/training is a fine-tuning service (LoRAQLoRASFT/

    DPO) wrapping axolotlunslothtransformers on a GPU pool. (The Stack is a fine-tuning + orchestration shop, NOT a foundation-model pretraining shop.)

  • Policy: self-hosted-first.kmd is explicitly a shadow-active / flip-on-gate

    model ("the kodec model: canonical Rust today, Koda shadow-active until it wins the benchmark"), with conjunctive gates per case-of-use (G2 perf ≤×1.05, G3 ≥3 stable releases).

3. Options

Description Long-term
a — wrap the mature ML stack (CUDAROCm + axolotlunsloth + Ollama/vLLM + frontier APIs) The current de-facto Best perf + full model ecosystem (Tensor Cores, optimized kernels). Self-hosted at serving; rents the kernels. Passes G2 today
b — Koder Rust-ML runtime (candle/burn over wgpu+CUDA) Vendor-neutral, Rust, Koder-owned The self-hosted compute endgame (not locked to NVIDIACUDA). But wgpu-compute << CUDA today (no easy Tensor Cores, model-support gap) → fails G2G3 now
c — own the kernels / Koda-ML Maximalist The ultimate own-your-primitives; competing with NVIDIA's kernel + Tensor-Core moat is a multi-decade endeavor — fails G2 by orders of magnitude. Not now

4. Decision: a canonical + b shadow-active (own the layers where competitive)

This is the self-hosted-first shadow-flip model, applied to AI compute:

  1. Canonical = a. Wrap the mature external GPU ML stack for the model math

    (fine-tuning: axolotlunslothtransformers; inference: Ollamallama.cppvLLM; frontier APIs for the hardest reasoning). It is canonical because it is the only option that passes G2 (perf) today — exactly what self-hosted-first prescribes ("the external remains default until the Koder option clears the gates").

  2. Shadow-active = b. The Koder-owned, vendor-neutral Rust-ML runtime

    (candle/burn) is a tracked maturation track — declared as a [self_hosted] pair (status experimental → shadow-active) so the interim never silently becomes permanent. It is benchmarked against the canonical per case-of-use and flips that case when it clears G2 (≤×1.05) + G3 (≥3 stable). Likely first-flip cases: small/specialized self-hosted models, embeddings, classifiers — where vendor-neutral Rust beats the CUDA dependency on ops/portability before it beats it on raw FLOPS.

  3. Own the layers where already competitive. The serving / orchestration / data

    plane — gateway, kdb-vector (retrieval/RAG), rag, memory, embed, the runtime — stays native Rust/Go, self-hosted (already done, passes the gates). This is where native+fast genuinely speeds up AI: lower latency/cost around the model (tokenization, batching, KV-cache, retrieval, routing).

  4. c (own kernels) = a far-future research horizon, not now.

5. Trade-offs accepted

Rent the GPU compute kernels (CUDA/external) at the model-math layer for the foreseeable future (vendor coupling there) — in exchange for best perf + the full model ecosystem now, while the vendor-neutral Koder compute matures as a tracked shadow that flips on evidence. Identical to kodec (canonical Rust, Koda shadow until it wins the benchmark).

6. Mechanism

  • The shadow pair is declared via a [self_hosted] block in the owning module's

    koder.toml (the self-hosted-pairs.md registry is generated from these). Seed: services/ai/runtime (the inference runtime) — see its backlog seed ticket.

  • Flip decisions are per case-of-use, gated by /k-bench G2 evidence +

    koder-spec-audit self-hosted --history for G3.

7. Orthogonality

This is independent of Kanvas (stack-RFC-019) — UI render vs model compute are different programs on the same hardware. Neither blocks nor accelerates the other.