Sandbox: runtime tiers + fleet-scheduled placement

Status

accepted — 2026-05-31 (owner-ratified). Drafted from a design conversation about whether the sandbox should integrate with infra/net/box (kbox) and infra/net/fleet; refined to the isolation-class lens (Axis 1). Ratified decisions on the open questions:

• Q1a — add the WASM class: YES. ~10 ms cold start is a genuinely new

  capability for short, frequent agent runs; serves the WASI-targetable subset (not a shared-kernel replacement). → SANDBOX-024 is unblocked.

• Q1b — shared-kernel slot: keep Incus for now. OCI is the same

  isolation class as the Incus system-container, so swapping is a benchmark-gated boot-latency optimization, not initial scope. Revisit only if a real-workload benchmark shows the boot win is material.

• Q2 — placement contract: sandbox CALLS Fleet's scheduler. The session

  stays a sandbox-owned object (tenantquotacapabilities/lifecycle); Fleet is a pure capacity/placement layer. NOT a fleet-native SandboxSession CRD. → SANDBOX-025 still needs a joint contract RFC with infra/net/fleet before implementation, but the direction + ownership split are ratified.

• *3 — WASM class capabilities: Exec + File IO + Upstream; no

  Terminal/Watcher*(no real shell to attach; Watcher is a full-OS concept).

• Q4 — sequencing: SANDBOX-024 (WASM driver) first, self-contained;

  SANDBOX-025 (fleet placement) after, gated on the joint fleet RFC.

The boundary contract + axes below are normative as of ratification.

Summary

Today a sandbox session is one Incus VM on one host (s.khost1), placed by a single in-process Manager warm pool. That fixes both how a session is isolated (one runtime) and where it runs (one host). This RFC separates those into two independent axes:

• Axis 1 — runtime tier (vertical: how isolated / how fast). The

  runtime.Runtime driver seam already abstracts this (subprocess / firecracker / incus). Add a kbox tier (OCI + WASM) so sessions can pick an isolation/latency class instead of always paying full-VM boot.

• Axis 2 — placement (horizontal: which host). Replace the single-host

  warm pool with Fleet-scheduled placement (infra-RFC-004), so the service spans a fleet of hosts — lifting the SANDBOX-019 single-host perf/pool ceiling and removing the SPOF.

Motivation

• Latency floor. An Incus system container boots in ~5–8 s. Many agent

  code-runs are short and frequent; a kbox WASM workload is ~10 ms cold (box#112, sandboxed by default). The current single tier overserves those.

• Capacity + SPOF. One host caps concurrent sessions and is a single

  point of failure. infra-RFC-004 reframes Fleet as a universal compute-orchestration backend that schedules workloads onto cluster nodes — exactly the placement problem sandbox hand-rolls today.

• The seams already exist. runtime.Runtime (driver plugin) and Fleet's

  scheduler make both axes additive, not rewrites.

Axis 1 — runtime tiers (by isolation class)

A runtime tier is not a linear "fast→slow" ladder; what actually differentiates a tier is where the security boundary sits. There are three real isolation classes, and a session's class is chosen by policy (language / workload / tenant plan):

Plus subprocess (no isolation) for dev/local only — not a production tier.

Key consequence (why OCI is not a separate tier): the Incus driver shipped so far runs system containers by default (cfg.VM gates --vm, default false). A kbox-OCI workload is the same isolation class as that — both are shared-kernel. So OCI and Incus-system-container are not two rungs; they are two implementations of the shared-kernel slot. Running both is double the driver surface (every capability × every driver) for zero isolation gain. The shared-kernel slot should have exactly one backing.

The optional capabilities (Terminal, Upstream, Archiver, Watcher, resource limits, network modes) are per-class — a class advertises what it supports; WASM omits the full-OS ones (terminal/watcher). SANDBOX-024 implements the WASM class behind runtime.NewKbox.

Axis 2 — placement

Today: Manager → local runtime.Runtime bound to this host's warm pool.

Proposed: Manager → a placement seam that asks Fleet to schedule the session onto a node, and dispatches the runtime.Runtime calls to that node's sandbox agent. Warm pools become per-node, fleet-aware; the global floor is a Fleet policy. SANDBOX-025 implements this.

Boundary contract (the crux)

Sandbox stays the tenant-facing API + policy layer; fleet is the placement layer; kbox is an artifact+runtime layer. A session is a sandbox concept that may be realized as a fleet-scheduled, kbox/incus-backed workload.

Open questions (for ratification)

• Q1 — which isolation classes do we back, and with what? Split into two

  sub-decisions, because OCI is not a separate tier (see Axis 1):

  • Q1a — add the WASM class? (recommend yes.) The ~10 ms cold start is

    a genuinely new capability Incus can't touch; the prize for short, frequent agent runs. Caveat: WASI is constrained (no full Linux syscalls; Python/Node-on-WASI immature), so it serves a subset of workloads — not a replacement for the shared-kernel class.

  • Q1b — who backs the shared-kernel slot: Incus (today) or kbox-OCI?

    *(recommend: keep Incus for now; treat OCI as a benchmark-gated boot- latency optimization, not initial scope.)* Both are the same isolation class, so this is a perf/ops trade, not an architecture one: OCI may boot faster (~100 ms vs ~5–8 s) and reuses kbox's image management (reuse-first), but Incus is already built + validated and upgrades to the dedicated-kernel class trivially (--vm). Only swap if a benchmark on the real workload mix shows the boot win is material.

• Q2 — placement contract shape. Does sandbox call Fleet's scheduler

  (sandbox owns the session object, fleet just places it), or does Fleet own a SandboxSession CRD (fleet-native)? The table above assumes the former (looser coupling); ratify the choice.

• Q3 — WASM class semantics. *(recommend: Exec + File IO + Upstream;

  omit Terminal + Watcher.)* WASM workloads are compute-oriented and short — an interactive PTY has no real shell to attach to, and Watcher is a full-OS concept. File IO (in/out of the WASI sandbox) and Upstream (if it serves HTTP) fit. This is exactly why capabilities are per-class.

• Q4 — sequencing. kbox driver (#024) and placement (#025) are

  independent; either can land first. Recommend #024 first (self-contained, no cross-component contract) to de-risk the tier model before the fleet coupling.

Alternatives considered

• Keep single-host + scale the host. Vertical-only; hits a wall and keeps

  the SPOF. Rejected as the long-term answer (still fine short-term).

• Sandbox builds its own scheduler. Duplicates Fleet (infra-RFC-004);

  violates reuse-first. Rejected.

• Add kbox-OCI as a third shared-kernel tier alongside Incus. Doubles the

  driver surface (every capability × every driver) for zero isolation gain — OCI and Incus-system-container are the same isolation class. Rejected as a tier; OCI is only in play as a possible replacement backing for the one shared-kernel slot, gated on a boot-latency benchmark (Q1b).

• Replace Incus with kbox-OCI now. Throws away a working, validated

  driver before a benchmark justifies it. Deferred, not rejected — revisit if Q1b's benchmark favors OCI.

Impact

• New: runtime.NewKbox driver (#024); placement seam in Manager (#025).
• Unchanged: the tenant-facing HTTP/WS API, the capability interfaces, the

  Incus driver (becomes one tier among several).

References

• infra-RFC-004-fleet-global-control-plane.kmd (Fleet control plane)
• infra/net/box (kbox; OCI + WASM), box#112 (WASM workload backend)
• sandbox-RFC-001-foundations.kmd (the sector's first RFC)
• SANDBOX-024 (kbox driver), SANDBOX-025 (fleet placement), SANDBOX-019

  (single-host perf ceiling), SANDBOX-014 (firecracker tier)

• policies/hyperscale-first.kmd, policies/reuse-first.kmd