Koder Runtime Strategy — kbox as the universal runtime

draft

Status

draft — opened 2026-05-20 from the kbox 10-wave session. Owner ratifies before Phase 0 work begins.

Update 2026-05-31: Phase 1 shipped (system containers + cgroup-v2 delegation, #129#138). Axis C (QEMU) re-ratified by owner into a 3-layer split: a shared runtime substrate (C.0, Phase 3.5, #134) consumed by both kbox (C.1 microVM dispatch) and a thin appliance-VM front-end (C.2, `infranetvm`, #132) — so the QEMU boundary integrates without duplicating kbox's authtracing/lifecycle plumbing. Rationale in Axis C below.

Summary

Replace Docker, LXC, and (where feasible) QEMUKVM across the Koder Stack with `infranetkbox` (kbox) as the universal containerruntime substrate. Three substitution axes with different gaps + phasing per axis. Explicit boundary: QEMU stays for Windows builds, kernel testing, and arbitrary-distro chroots — kbox does not try to be a full hypervisor.

Motivation

Today the Stack runs four different runtimes for overlapping problems:

Tech Where Pain
Docker s.poc.vivver.com (4 containers), CI runners, dev One more dependency to keep current; supply-chain (Docker Inc) outside the Stack
LXC/Incus s.khost1 (5 CIMMG LXCs + cert daemon + Koder VPN hub) Different mental model than Docker; per-host install + CLI culture; not aligned with Koder Stack tooling (no Jet integration, no Koder ID, no shared image layer cache)
QEMU/KVM Win10 VM + kodix chroot + Koda toolchain VM (planned) Necessary for the cases that need it; no Stack-side abstraction over libvirt
(cosign etc.) tooling around runtime —

Convergence on one runtime — kbox — gives uniform multi-tenancy (#108), uniform tracing (#109+#122), uniform OAuth (#110), uniform signing (#114+#121), one set of dashboards, one runbook, one tooling install per host. This RFC says how that convergence happens without losing the cases QEMU genuinely covers.

Three substitution axes

Axis A — Docker → kbox

Gap: smallest. kbox v1.1.0 has OCI runtime + image builds + registry + compose + CRIU + farm. Waves 1-10 added the Stack-native pieces (TOML config, multi-tenancy, tracing, OAuth, signing, OTLP, SBOM, quotas). Pair #13 in self-hosted-pairs.md is at stable for single-node; the open gates are operational, not feature.

What's missing for production_proven: real deployment + soak + Hub publication + bench numbers. Code-complete.

Axis B — LXC → kbox

Gap: medium. LXC runs system containers (full OS userland + systemd + persistent multi-service). kbox today runs application containers (single-process tree). Translating Incus workloads (5 CIMMG LXCs) requires kbox to grow:

  • kbox run --system (PID 1 = systemd, runs forever, multi-process)
  • kbox attach <id> (shell into running system, distinct from exec <id> <cmd>)
  • kbox profile create/apply (Incus-style reusable config template)
  • kbox snapshot create/restore (disk-level snapshot, distinct from CRIU live-migration)
  • Stateful disk pool UX matching lxc storage create

None of these are blocked technically — they're feature work not yet on the backlog.

Axis C — QEMU/KVM → kbox

Gap: largest, partially out-of-scope. Three workload classes use QEMU today:

  1. Windows guest (Flutter Windows build via k.win WinRM) — kbox cannot run Windows. Hard boundary.
  2. Kernel testing / kodix ISO chroot — needs separate kernel. Hard boundary unless kbox grows a microVM dispatch.
  3. Strong-isolation Linux workloads (Koda toolchain VM planned per policies/test-host-isolation.kmd) — these CAN move to microVM if kbox grows the dispatch.

Decision: split this axis by workload contract, not by "container vs VM" (owner-ratified 2026-05-31).

The deciding lens is the workload's contract, not whether it's "a container" or "a VM":

  • OCI run-to-lifecycle (image → boots → runs → exits): namespaced containers AND Firecracker microVMs share this exact contract — same kbox run <image> semantics, only the isolation differs. → kbox backend (--runtime=namespace|microvm).
  • Stateful appliance VM (Windows guest, arbitrary-kernel test, cross-arch emulation): own disk image, BIOSUEFI, consoleRDPVNCWinRM, snapshot-as-disk. This does NOT fit kbox run <oci-image> — it is closer to a persistent appliance than to a run-to-completion workload. → a distinct front-end, NOT a kbox backend.

This yields three layers (rejecting both "fold everything into kbox" — which makes kbox a god-object spanning two disjoint data models — and "standalone VM module that re-implements the plumbing" — which duplicates authtracinglifecycle, violating reuse-first.kmd):

  • C.0 runtime substrate — the cross-cutting host-workload plumbing both fronts share. A reuse-first.kmd walk of engines/sdk/go/ (SDKGO-005, 2026-05-31) found this is mostly already shared SDK — there is no broad extraction to do:
    • Auth ✅ already engines/sdk/go/auth (JWKS validator + middleware); kbox already consumes it (internal/server/oauth.go is a thin adapter).
    • Observability ✅ already engines/sdk/go/obs (OTLP + spans + RED + structured logs, adopted by koder-notifysymbol-store). kbox still runs a bespoke `internaltracing/ → migrate kbox onto obs` (BOX-140); not a new clone.
    • Multi-tenancy — the contract is specs/multi-tenancy/; kbox's internal/server/tenant.go is small + container-specific → stays local.
    • Lifecycle verbs + host-instances registry — the only genuinely-new shared piece, but it has 1 consumer today (kbox); the VM module doesn't exist yet. Per reuse-first.kmd (promote on the 2nd consumer), it is extracted with BOX-132, co-designed against its real 2nd consumer — not pre-emptively.
  • C.1 microVM dispatch — kbox grows a Firecracker backend for OCI-shaped Linux workloads wanting kernel isolation. Same UX as kbox run, just --runtime=microvm. kbox consumes C.0. Covers the strong-isolation Linux class.
  • C.2 permanent appliance-VM scope — Windows + kernel testing + cross-arch stay in QEMU forever (intentional boundary, not a gap). A thin infra/net/vm/ front-end wraps libvirtQEMU and consumes the same C.0 substrate — so it is NOT a copy of kbox's plumbing, just a different door for a different workload shape. Net effect: separate binaries only where the workload shape genuinely differs (neither program rots), with zero duplication of authtracing/audit and a single "what's running on this host" view spanning both containers and VMs.

Phasing

Phase Scope Owner-side gate Estimated duration
0 — Docker production_proven Replace Docker on s.poc.vivver.com (4 containers). Land bench numbers in perf-baseline.md. Publish to Hub. First external deployment + 7-day soak ~1-2 months
1 — System mode Add kbox run --system + attach + profile + snapshot. Pilot: replace cert daemon LXC on s.khost1. Cert daemon (already stable per project_certs_deployed_skhost1) runs in kbox-system for 7 days without regression ~2-3 months
2 — LXC at scale Migrate CIMMG LXCs (db, app, neogestor, tv-fila, esus) + Koder VPN hub. All 6 LXCs migrated; Incus dependency removable from s.khost1 (kept as fallback) ~1-2 months
3 — MicroVM dispatch Add Firecracker backend to kbox. kbox run --runtime=microvm. Target: Koda toolchain VM per test-host-isolation.kmd. Toolchain VM runs in kbox/Firecracker without regression vs raw QEMU ~3-4 months
3.5 — Substrate audit (done) + obs adoption SDKGO-005 walk: auth already SDK (adopted), tenancy stays local, lifecycleregistry deferred to Phase 4 (1 consumer today). Residual: kbox adopts `enginessdkgoobs, retiring bespoke internaltracing` (BOX-140). kbox on obs; OTLP E2E re-validated to Tempo; internal/tracing deleted ~3-4 days (was over-scoped at ~3-4 weeks)
4 — Appliance-VM boundary Document permanent QEMU scope (Windows + kernel test + cross-arch). Thin infra/net/vm/ front-end over libvirtQEMU consuming the C.0 substrate (no re-implemented authtracing). Win10 VM + kodix ISO chroot still in QEMU but auth+tracing flow through the SAME substrate as kbox; one "what's running here" view spans containers+VMs ~1 month

Total elapsed: ~8-12 months, depending on parallelism with other Stack waves.

Per-phase implementation tickets

Opened against infra/net/kbox/backlog/pending/:

  • #128 Phase 0 — Docker production_proven activation (deploy s.poc.vivver.com, soak, bench, Hub publish)
  • #129 Phase 1 — System container mode (--system + attach + profile + snapshot)
  • #130 Phase 2 — CIMMG + cert-daemon LXC migration
  • #131 Phase 3 — Firecracker microVM dispatch backend
  • SDKGO-005 (enginessdkgo) Phase 3.5 — DONE (audit): reuse-first walk found no broad substrate to extract (auth already shared, obs available, lifecycle/registry deferred). Spun off BOX-140.
  • #140 (infranetkbox) — kbox adopts engines/sdk/go/obs, retires bespoke internal/tracing/
  • #132 Phase 4 — thin infra/net/vm/ appliance-VM front-end; reuses engines/sdk/go/{auth,obs}; extracts the lifecycle/host-instances registry into the SDK as part of building this module (co-design with its 2nd consumer, per reuse-first)

Out of scope

  • Windows guest in kbox. No. Stays in QEMU.
  • Replacing Incus itself. Incus stays installed on s.khost1 as fallback during Phase 1-2 migration; removable only after Phase 2 completion.
  • Cross-architecture emulation (arm guest on amd host etc.). Edge case — handled by QEMU when needed.
  • Live container migration across hosts. kbox has CRIU local-host migration already; cross-host depends on kdb cluster state (kbox#115, blocked on kdb#414).

Open questions

Q1 — System-mode default for attach

LXC's attach enters the container as root with a fresh login shell; Docker's exec runs a new command in an existing namespace. For kbox attach, do we mimic LXC (fresh-shell semantics) or just alias to exec /bin/sh?

Recommendation: Mimic LXC (fresh login session, reads /etc/profile). Operators migrating from lxc shell expect this. Add kbox exec for the Docker-style behaviour as already shipped.

Q2 — Single Stack-wide runtime config or per-host?

Today kbox.toml is per-host. For multi-host Stack management (Koder Console fleet view), do we want one global runtime policy distributed via kdb, or stay per-host?

Recommendation: Stay per-host for Phases 0-2. Revisit when kbox#115 (cluster state) ships.

Q3 — Firecracker vs Cloud Hypervisor vs Kata

Three open-source microVM options. Trade-offs:

  • Firecracker — minimal, AWS-backed, well-documented; Linux x86_64 + arm64 only
  • Cloud Hypervisor — Intel-backed, similar surface, slightly heavier
  • Kata Containers — full OCI-shim layer over Firecracker/CH; more abstraction, less control

Recommendation: Firecracker. Smallest surface area, longest production track record, simplest integration. Re-evaluate if a specific limitation bites (no GPU passthrough is the most-cited; if Koda toolchain testing needs GPU we'd revisit).

Q4 — Incus removal threshold

After Phase 2, when is it safe to apt remove incus from s.khost1? Risk: a future LXC need surfaces that kbox-system hasn't covered yet.

Recommendation: Keep Incus installed but inactive (no running containers) for 90 days post-migration. Hard removal after 90 days with no rollback request. Document the rollback procedure (re-create containers from snapshots) in infra/net/kbox/docs/migration-from-incus.kmd.

Acceptance criteria for this RFC

  • Owner ratifies → Phase 0 work starts on s.poc.vivver.com
  • 5 tracking tickets opened in infra/net/kbox/backlog/pending/
  • registries/self-hosted-pairs.md row 13 references this RFC in the notes column
  • meta/docs/stack/modules/box.md gains a "Runtime strategy" section pointing at this RFC + the phasing

Cross-references

  • registries/self-hosted-pairs.md row 13 (Kbox) — current gate state
  • policies/self-hosted-first.kmd — the gates this RFC closes
  • policies/test-host-isolation.kmd — Phase 3 / 4 boundary (test workloads in VMs)
  • infrastructure/servers.md — list of hosts using each runtime today
  • infra/data/kdb#414 — blocks Phase 2.5 multi-host clustering (kept separate from this RFC)