Box — OCI Container Runtime

• Area: infra/net/
• Path: infra/net/kbox
• Slug: kbox (alias kbox)
• Kind: OCI-compatible container runtime + image manager (Go)
• Version: v1.1.0 (CHANGELOG: infra/net/kbox/CHANGELOG.kmd)
• Self-hosted pair: row 13 of registries/self-hosted-pairs.md — replaces DockerPodmancontainerdbuildahnerdctl. Status: stable for single-node runtimebuildregistrycomposeCRIU; experimental for clusterCDIWASMCVEICP-Brasil signing.

Role in the stack

Box is the Koder OCI container runtime — the layer that actually runs containerized applications on top of infra/linux/kodix. Every Koder service or product packaged as an OCI image (whether deployed via infra/net/cloud or a manual server provision) ultimately runs inside Box.

It is the substrate for multi-tenant compute: namespace isolation, cgroup resource limits, lifecycle management, image pulling, layer storage, volume mounts. v1.1.0 added live checkpointrestore (CRIU), distributed build farm, and modern rootless networking (pasta, macvlanipvlan, netavark/aardvark-dns).

Surfaces

Configuration

/etc/kbox/default.toml (TOML) → KODER_BOX_* env overrides → CLI flags. The TOML loader landed in #104 (Go 1.24+, BurntSushi/toml); the schema mirrors the JSON-tagged internal/config.Config. Missing file is silent (defaults kept); malformed file is a startup error.

Multi-tenancy

Every container carries a KoderUserID field and an OCI label dev.koder.tenant.user_id (mirrors). The REST API resolves the tenant from the X-Koder-User-ID request header (until OAuth Koder ID daemon auth lands — ticket #110); listinspect handlers filter by tenant, and cross-tenant lookups return 404 (not 403) per `specsmulti-tenancy/contract.kmd. Legacy containers without the field resolve to the special tenant unknown` so they remain visible to administrators during the migration window. Landed in #108.

Primary couplings

k-evolve 2026-06-06 — gap analysis of the mature module (147 done) opened #163–#169 (verified, evidence-anchored): #163 daemon request body-size limits (DoS hardening — DONE* bodyLimitMiddleware), #164 wire quota.Manager into create (per-tenant fairness; quota exists since #125 but was never called on create), #165#166#169 unit-test the untested packages (internal/volume, internal/compose, internal/{quadlet,kube,remote} had 0 _test.go), #167 API list pagination, #168 --gpus CLI alias → CDI. (A stale "OTLP exporter missing" gap was rejected — kbox already exports OTLP/gRPC via obs.SetupOTLP, box#140.)

infra/net/kbox/backlog/pending/ — Phases 0234 of RFC-001 (#128 stale-premiseowner re-scope, #130-#132; Phase 1 done) and #149 (monitor-owned log framing, low-pri follow-up to #146). Shipped 2026-06-01: #146 (conmon-style detached monitor), #147 (NetworkPolicy nftables dataplane), #148 (per-PID resource stats) — see sections above. Shipped 2026-06-03: #133 (multi-jurisdiction sign --jurisdiction consumer — see section below). The done/ directory contains 30+ tickets from waves 1-10 (TOML config, multi-tenancy, tracing+OTLP, OAuth+JWKS, CDI, self-update, ICP-Brasil signing sign+verify, SBOM, CVE scan, WASM, per-tenant quotas, pull verify policy) + Phase 1 system-container mode (#129#138) + image-tooling fixes (#139: importcommit materialize layers; flat-rootfs import) + one-shot output capture #144 (detached kbox run -d output reached an empty kbox logs because the in-process capture goroutine died with the CLI; fixed by writing the container's stdoutstderr to the log fd directly — kernel-written, survives the detached CLI exit), #145 (kbox run silently dropped a -flag in the container command, e.g. sh -c '…''s -c), and #143 (in-container signal to a host PID validated against the container's cgroup membership — no host-PID escape; kbox kill --pid + `POST /containers{id}signal; the exec-over-API half already existed), and #142 (per-container process-event stream + enriched process snapshot + structured cgroup-stats, sourced from cgroup.procs deltas — kbox container-procs [--follow] + GET /containers{id}{processes,process-events,cgroup-stats}`, consumed by Fleet FLEET-023025).

Unified host-instances view — hostinst repoint (#152, done 2026-06-02)

kbox now implements the shared engines/sdk/go/hostinst contract (infra-RFC-001 §Axis C, the C.0 substrate extracted by Koder VMBOX-132 as the 2nd consumer — reuse-first "promote on the 2nd consumer"). `internalruntimehostinst_adapter.go exposes Executor.AsHostInstBackend() implementing hostinst.Lister+Lifecycle: List maps each core.Container→hostinst.Instance` (KindContainer, coarse state, tenant label), StartStopDestroy delegate to the existing executor verbs (no forked logic). A hostinst.Registry provably merges kbox containers + Koder VM domains into one sorted host view (tested with a stub VM backend). Purely additive — kbox ps and the BOX-146 monitor are untouched. The cross-binary kbox ps --all-runtimes that discovers the real VM backend was split to #157 — DONE 2026-06-06: discovery is registry-over-socket (option 2, Quality>Speed) with a kvirt list --json shell-out fallback, both behind hostinst.Lister so socket↔exec swap is transparent and no libvirt couples into the kbox binary. New SDK hostinst.SocketListerExecLister (engines/sdk/go/hostinst/discovery.go); the kbox daemon serves GET /api/v1/host-instances; kbox ps --all-runtimes (in-process kbox + vm socket→kvirt-fallback, KIND column) and the symmetric kvirt list --all-runtimes (in-process libvirt + kbox SocketLister). Graceful degrade when a runtime is absent; kbox ps (no flag) untouched.

Host-level checkpoint for power-loss resilience — suspend-host / restore-host (#154, implemented 2026-06-02)

The per-container CRIU primitives (#092#093#094) and the cgroup freezer (#023) freeze one container; #154 adds the node-level emergency operation for the imminent-powerhardware-failure scenario — freeze the whole box to durable storage at once, power off safely, resume from the exact pre-suspend point on the next boot. kbox suspend-host enumerates every running container and, bounded-parallel, CRIU-checkpoints each (best-effort) with a graceful flush-stop fallback for workloads CRIU can't dump (host-netns, GPU fds), then sync(2)s and writes a self-describing suspend-manifest.json; it exits power-off-safe only when every container is checkpointed or flushed-stopped (a durability guard refuses a tmpfs target). kbox restore-host (a kbox-restore-host.service oneshot, Before=kbox.service) replays the manifest — CRIU-restores checkpointed containers (checksum-verified, new id remapped to workload identity), cold-starts flushed ones — and is *dempotentcrash-safe (per-entry progress persisted, so an interrupted restore re-runs without double-restoring). Orchestration lives in internal/hostsuspend against an inverted ContainerOps interface (no internal/runtime import → the Fleet-facing server API can drive the same path for fleet#043). Implemented + unitbuildvet-verified on dev-linux-box. The #155 live drill (criu 4.1.1) then verified the orchestrationfallbackmanifest/safe-exit path end-to-end on a real container — but surfaced that the CRIU checkpoint path itself is broken against a kbox container (a latent #092/#093 defect, never validated against kbox's unshare --pid shim): the dump targets the outer shim pid not the container-init pid-1, the init isn't a session leader in its pidns, and the netns dump needs iptables. #156 (P1) fixed the DUMP half (validated live): the __shim__ now setsids non-TTY container inits and internal/criu dumps the inner-pidns init (not the outer shim pid) — kbox suspend-host now reports 1 checkpointed with a complete 34-image dump (iptables was already in the deb Depends; the drill error was a VM artifact). The RESTORE half is the full container-CR restore stack (bisected on the live VM): cgroup-flag mirror ✅, netns via --empty-ns net ✅ (the inline netns restore fails on userns sysctls), but criu then segfaults restoring the mount namespace*(`BUG at criumount.c:48) — kbox's pivot_root+overlay+userns mounts need --root+--external mnt[] coordinated dump↔restore (the OCI→criu mount mapping runc builds via CRIU swrk/RPC). This is a substantial dedicated effort, not CLI-flag tweaks — now tracked as its own ticket **#158** (restore mount-namespace C/R via CRIU swrk/RPC: overlay rootfs re-prep + external mount map co-declared dump↔restore), after **#156 (the dump fix) closed done 2026-06-03**. A **2026-06-03 live bisection** (dev-linux-box, criu 4.1.1) peeled the restore mount stack **four layers** deep: (1) mount.c:48 segfault → --external mnt[dev]:kdev; (2) mount-v2.c:628 → re-mount the overlay rootfs before restore; (3) mount.c:2256 overlay-in-userns EINVAL via the new mount API → **--mntns-compat-mode` REQUIRED*(forces the legacy mount engine — the key non-obvious finding); (4) host devtmpfssysfs external bind EINVALs in userns. A 2nd live session (dev-linux-box confirmed a VM → the EINVAL is FUNDAMENTAL to overlay+userns, not a nesting artifact) found /dev+/dev/shm externalized bind OK but /sys (sysfs-userns) is the blocker — and critically that CRIU's CLI mount handling is non-deterministic w.r.t. the external set (adding /sys changed whether /dev binds), confirming the CLI cannot express the precise per-mount control needed. Decision: implement via swrk/RPC (go-criu). #158 LANDED 2026-06-03 — internal/criu/swrk.go drives criu via go-criu RPC with the validated recipe (re-mounted-overlay Root + MntnsCompatMode + EmptyNs net + ManageCgroupsMode IGNORE + a FRESH dev tmpfs populated with device nodes + fresh /devshm; /sys criu-native = fresh sysfs). Live proof: kbox checkpoint→kbox restore resumes a real container with RAM preserved (count-continuity 6→10, no reset). The runtime-owned hardening is split to #159 — items 1+2 DONE 2026-06-03: (1) cgroup re-attach (Executor.ReattachRestoredCgroup + CheckpointMeta.Resources → a --memory 64m container restores with memory.max=64m); (2) monitor adoption + log continuity — kbox restore now spawns a __restore_monitor__ that drives the criu restore with inherit_fd reconnecting the restored stdoutstderr onto the new monitor's #149 framer pipes (post-restore output reaches kbox logs), Wait4-supervises the RstSibling child, and reaps on kbox stop exactly like a fresh container. The inherit-fd recipe (record fd1fd2 pipe:[<inode>] at dump; clear O_CLOEXEC on the inherit fd since go-criu's forked criu swrk inherits by number) was live-validated. Item 3 (bridge-net veth re-wire) DONE 2026-06-03 (adcaeeace5): Executor.ReattachRestoredNetwork re-creates the veth + re-assigns the recorded IP into the fresh --empty-ns net netns — a bridge container restores with eth0 UP at its same IP + 0% loss to the gateway. Item 4 (TCP survival) DONE 2026-06-03 (a1c47ed833): the network is now configured DURING restore via the criu SetupNamespaces Notify hook (the post-restore re-wire was too late — criu restored the socket into the empty netns → soccr: Network is unreachable); an established TCP connection now survives checkpoint→restore (validated: stream continues after restore). Also fixed a real IP-pool collision bug (2edb50b0ab: every container got the same IP because the in-memory pool wasn't seeded from running containers). Item 5 (full #155 reboot-resume drill) DONE 2026-06-06 → #159 + #155 CLOSED. The live drill is GREEN on dev-linux-box (criu 4.1.1) via a real incus restart: kbox suspend-host → 1 checkpointed, 0 flush-stopped (real dump) → boot-time kbox-restore-host.service Restored 1/1 unattended (exit 0) → an in-memory counter journalled to the rootfs (/ticks) read …23 [reboot] 24 25…61 — perfectly contiguous, no rewind/gap = RAM preserved across a real reboot (restored process ELAPSED ≫ time-since-boot corroborates); re-running restore-host is idempotent (no double-restore). The kbox RAM-preserving release is now unblocked, and fleet#043 can fan out suspend-host per node. Drill procedure + evidence: tests/regression/BOX-155-suspend-host-reboot-resume.md (registry #901). #161 (the boot-path log-capture gap) DONE 2026-06-06: the boot restore-host now routes through the same monitor-owned restore as kbox restore (a new inverted ContainerOps.RestoreCheckpointed verb → Executor.RestoreCheckpointed→StartRestoreMonitor), so a boot-restored container gets kbox logs (inherit_fd) + bridge-net + cgroup + supervision parity — not a bare inline criu.Restore. Live-validated: suspended at tick 11 → real reboot → boot-restored kbox logs flowed from tick 12 (was empty before). So per-container checkpoint→restore is now production-complete: RAM-preserving + cgroup limits + logs/lifecycle + bridge networking (same IP) + established-TCP survival. Also fixed this session: the kbox restore <name> name→container-ID resolution bug (checkpoints are keyed by full ID; restore now resolves via GetContainer like checkpoint does). Scope reality: box#154's primary guarantee — power off safely, no data loss — already holds via the flush-stop fallback; CRIU RAM-preservation is the upgrade, so the restore-half is non-urgent and the release is held only on the RAM-preserving claim, not the safety floor. This is the node-local foundation fleet#043 fans out across a cluster on a UPS signal.

Detached supervision — conmon-style monitor (#146, done 2026-06-01)

A non-TTY kbox run -d (detach is the default) prints the container id and the CLI exits immediately, which used to kill the in-process cmd.Wait() goroutine before a fast one-shot was reaped — losing the exit code, leaving state stuck running, and deadlocking kbox wait. kbox now interposes a persistent kbox __monitor__ <container-dir> process (mirroring Dockerpodmancri-o's conmon): a Setsid-detached supervisor that becomes the container's parent for its whole lifetime, reaps it, writes the exit code to a durable exit file, transitions on-disk state, and applies the restart policy — independent of the launching CLI. spawnProcess hands the fully-resolved launch (shim + aa-exec wrap folded into argv; ambient caps / credential / cgroup-path carried as data in launch.json) to the monitor and gets the pid back over a handshake file, then does the pid-dependent setup (cgroup placement, networking) as before. Wait() was rewritten to poll the durable record (exit file → on-disk state → dead-pid guard) so it can never deadlock; StopKill write a stop-requested marker so the monitor steps aside (no restart, no clobber). TTY containers keep their own in-process handler. Escape hatch KBOX_NO_MONITOR=1. Verified on dev-linux-box (exit codes 730 incl. the instant-exit race, no kbox wait deadlock). The self-exec entrypoints (__shim____monitor____pause) live in shared `internalruntimeselfexec.go and are dispatched by **both** the kbox CLI and the kbox daemon — BOX-150 fixed a regression where the daemon dispatched none, breaking daemon container start for the default (apparmor + caps-dropped → shim) profile. BOX-151 then fixed the next layer: server.NewServer` built the imagenetworkvolume stores but never wired them onto the executor, so daemon containers got an empty rootfs (`stat /binsh: no such file) — now wired (+ LoadState). With both fixed, the FLEET-040 NetworkPolicy chain is proven live end-to-end on a real daemon container (verdict POST → nftables in the container netns + clear). The optional JSON-lines framing / stdout-stderr split / live kbox logs -f` via the monitor is deferred to #149.

NetworkPolicy dataplane enforcement (#147, done 2026-06-01)

kbox owns the container netns, so it owns the kernel-level half of Fleet's cross-namespace isolation. internal/network/netpolicy.go programs an inet kbox_netpol nftables table in the pod's netns from a resolved PolicyVerdict (per-governed-direction base chain with policy drop, ct-established + loopback admits, one accept per allow rule, dual-stack), via ApplyPodNetworkPolicy(pid, verdict) / ClearPodNetworkPolicy(pid). The single source of truth for evaluation stays in Fleet (FLEET-030 PolicyEngine); kbox exposes a PolicyResolver seam (Manager.WithPolicyResolver) that Fleet plugs into, handing down a verdict with peer selectors already expanded to concrete CIDRs. Teardown is automatic (the table dies with the netns). Verified on dev-linux-box (real netns: baseline ping → default-deny DROP → explicit-allow OK). The Fleet-side adapter (selectors→CIDRs + inject resolver) is tracked in FLEET-040. The receiving endpoint POST /containers/{id}/network-policy decodes a verdict and calls ApplyPodNetworkPolicy(c.Pid, verdict) — the sink Fleet's resolver (FLEET-040: PolicyEngine.ResolveVerdict + KboxClient) POSTs to; both sides' JSON tags are pinned identical. The full Fleet→kbox e2e is gated on Fleet gaining a real (non-simulated) runtime executor.

Per-PID resource stats (#148, done 2026-06-01)

The BOX-142 process inventory now carries per-PID utime_ticksstime_ticks (USERHZ; ClockTicksPerSec=100) + `rssbytes, read from procpidstat for the PIDs in the container's cgroup subtree (ContainerProcesses / GET containers{id}processes / kbox container-procs`). Argv stays hashed (privacy-preserving). Lets Fleet's FLEET-025 telemetry attribute CPUmem to a named process, not just the workload cgroup tier.

Multi-jurisdiction image signing (#133, done 2026-06-03)

kbox image sign --signer koder delegates to services/crypto/signer for legally-valid signatures attached as OCI manifest annotations. --jurisdiction selects the legal profile and accepts a CSV (br,eu,us) — kbox signs once per jurisdiction, writing one envelope each under a distinct annotation key (dev.koder.signature.{icp-brasil,eidas,esign}), so a single image carries several signatures (RFC Q2 multi-envelope). The driver never short-circuits: a profile the signer hasn't promoted yet (euus today → HTTP 501 KSIGNER-JURIS-6001) is skipped with a warning at exit 0, so the br envelope is still produced; --strict turns any failure into a hard error. Signer KSIGNER-* codes are translated into a one-line message naming the wave that unblocks each (SignerError + TranslateSignerError in `internalimagebox133multijurisdiction.go). kbox image verify --signer koder --jurisdiction j reads the matching annotation back. Only br (ICP-Brasil) is live; eu`us plumbing is proven byte-for-byte by an httptest mock and will work unchanged the moment the signer promotes them (signer#014 stages 2b-22c3 + wave F remain on the signer backlog, not box's). Self-hosted pair status accordingly upgrades from experimental toward stable for the consumer-side signing surface.