Fleet as a Global, Federated Control Plane

Document format reference: metadocsstackpoliciesdocument-format.kmd

Status

Ratified 2026-05-31 (owner). All five §6 decisions resolved. The single-cluster Fleet (FLEET-011..022) is the foundation; this RFC defines the multi-host / global evolution (capabilities G1–G7). The gate is G1/FLEET-028 (durable state → kdb). Implementation is a multi-epic campaign sequenced in §8.

Implementation status (2026-06-01): the buildable campaign is complete — G1 (FLEET-028), G2 (031), G3 (032), G4 (033), G5 (034), G6 (035), G7 API+profiles (036) all DONE. Remaining: G6b/active-active+cross-region+mTLS (FLEET-037, gated on kdb-RFC-008 multi-region live + a real 2nd region) and the G7 UI (FLEET-038, Flutter web). Cross-host kbox-dump shipping in true multi-node = node-agent proxy (FLEET-027). See meta/docs/stack/modules/fleet.md.

1. Motivation

Today Fleet is a single-cluster orchestrator: one control plane, one in-memory ResourceStore, scheduling pods onto the nodes of that cluster. The owner's vision is broader: Fleet as a universal compute-orchestration backend that works at three scopes with one API + one UI —

1. Local cluster — one host (e.g. s.khost1), today's scope.
2. Global / federated — many hosts across sites, Fleet placing a VM/kbox on

   the most suitable host, migrating kboxes between hosts for resource rebalancing.

3. Managed clusters as a product — a requester asks Fleet not just for a

   VMkbox, but for a whole managed cluster of VMskboxes (on one host or spread across hosts) to test HA, load-balancing, and failure modes of the programs they are developing.

…serving both machine consumers (API, automated orchestration) and humans (UI), for both Koder Stack customers and Koder collaborators (with a profile that grants resource monitoring + management).

This is the Kubernetes-federation / Cluster-API problem, built native on the Koder substrate (kbox runtime, lease lifecycle, kdb data plane, Koder ID auth).

2. The single hard blocker: state

The current internal/core.ResourceStore is in-memory (a map guarded by a RWMutex). Everything global depends on replacing it:

• R-STATE-1. Cluster state MUST move to a persistent, consistent backend.

  Per stack-RFC-001 (kdb as the unified data plane) the default is kdb-next (or pkg/kdb/PG with RLS today). The store interface stays; the backend swaps. This is the prerequisite for HA control-plane (FLEET-019 leader election only becomes real once the lease + state survive a process, which needs durable storage + quorum — see FLEET-002).

• R-STATE-2. Until R-STATE-1 lands, "global" is impossible: two control

  planes cannot coordinate over a per-process map. Sequencing: state backend first, federation second.

3. Architecture — three tiers

            ┌──────────────────────────────────────────┐
            │  Global control plane (Fleet "hub")       │
            │  - durable state (kdb-next, R-STATE-1)     │
            │  - global scheduler (host selection)       │
            │  - federation registry (member clusters)   │
            │  - one API + one UI (RBAC by profile)      │
            └───────────────┬───────────────┬───────────┘
                            │               │
              ┌─────────────▼──┐      ┌──────▼─────────────┐
              │ Cluster A (host)│      │ Cluster B (host(s))│   ← member clusters
              │  Fleet control  │      │  Fleet control     │     (today's single
              │  + node agents  │      │  + node agents     │      cluster = a leaf)
              │  → kbox runtime  │      │  → kbox runtime    │
              └─────────────────┘      └────────────────────┘

• A member cluster is exactly today's Fleet (single control plane + agents

  over kbox on one or more hosts). Nothing about the leaf changes conceptually.

• The global hub is a new role of the same binary: it holds the federation

  registry, runs the global scheduler, and proxies/aggregates the API + UI across members. A single-host deploy runs hub+member co-located (no new operational burden for the local case — the "global" tier is opt-in).

4. Capabilities this RFC introduces (each a future ticket/epic)

5. What already exists in the Stack (so this is wiring, not greenfield)

• kbox (infra/net/box) — the runtime + CRIU checkpointrestoremigrate

  (the mechanism G3 needs) + per-host process/cgroup visibility (BOX-142).

• koder-lease (infra-RFC-003) — ephemeral-compute lifecycle (TTL, idle reap,

  saturation-aware admission). The global scheduler's elasticity layer.

• kdb / kdb-next (stack-RFC-001) — the durable, multi-tenant (RLS) data plane

  for G1.

• Koder ID (specsauthoauth-flow) + Fleet RBAC (FLEET-017/020) — the

  profile-based authz for G7 (customer vs collaborator).

• Koder Observability (infra/observe) + FLEET-025 — per-host/-workload

  telemetry feeding the rebalancer (G4).

• Fleet single-cluster core (FLEET-011..022) — the leaf that federates.

6. Decisions deferred to ratification (open questions)

1. Consistency model of the global hub — RESOLVED (2026-05-31). Separate the

   two planes; do not conflate them:

   • Data plane (Fleet state) → delegated to kdb. Fleet's Store (G1 /

     FLEET-028) is the only persistence; Fleet writes zero replication code. Multi-region of the data plane is therefore inherited from kdb: single- region today, geo-replicated the moment kdb's RFC-002 (multi-region) lands — "without schema changes" per kdb-RFC-001 Goal 3. self-hosted-first + reuse-first: the Fleet must never grow its own replication layer that duplicates (worse) what kdb is building.

   • Geo-readiness from day 1 (FLEET-029). Every Fleet API + stored object

     carries a region tag from the start (mirrors kdb-RFC-001 Phase 5 "geo- readiness audit"). Near-zero cost now; makes multi-region *additive later without a schema break*. This is the concrete "implement the readiness, not the feature now" step.

   • Control plane (scheduler/reconcile leadership) → active-passive, leader

     per scope via FLEET-019, independent of how many regions kdb spans. Active-active multi-region of the hub (two hubs reconciling the same cluster concurrently) is gated, not on timidity but on two real dependencies: (i) kdb-RFC-008 geo being live, and (ii) a genuine 2nd region + the latency/ conflict/jurisdiction requirements that only real cross-region load defines. Distributed data + centralized-with-failover decision is the safe shape; active-active control planes are where split-brain lives. Revisit per always-on.kmd when (i)+(ii) hold.

> Decision 2026-06-01 (owner, via /k-go fleet) — mature now, don't scramble later.

> The original gate conflated two things; we split them so maturity isn't deferred to

> customer-pressure time:

> - *A) Buildable-now (no real customer / no real 2nd region needed — matures against a

> standby/simulated 2-region substrate):*kdb-RFC-008 single-primary geo (phases

> 8.1 region metadata + mTLS, 8.2 WAL replication, 8.3 read-routing, 8.4 automatic

> failover) and, on the Fleet side, hub↔member mTLS + *egion-complete inherited

> state (→ FLEET-039*. These are real engineering against a second real kdb

> instance (two region tags, netem-injected partition/latency), not theater — so we build

> and chaos-test them now to reach production maturity before the first international

> customer. Substrate: a locally-simulated 2-region setup on s.khost1 (two region-tagged

> LXCs) is the $0 build+chaos substrate; the GCP koder-fleet-geo-us node is right-sized

> on-demand only for occasional real-WAN validation passes (the sole $ decision, deferred

> to the validation phase).

> - (B) Genuinely gated (correctness is empirical — needs real cross-region load):

> active-active multi-primary (two hubs / two kdb primaries on the same shard) — this

> is kdb RFC-009/CRDT (RFC-008 §8.5), beyond single-primary geo. Stays in FLEET-037,

> gated as before.

> kdb-side execution tracked as a carve-out anchor in the kdb backlog (the kdb component is

> under a concurrent session lock; RFC-008 itself is edited by that owner, not here).

2. Migration scope. kbox (CRIU) is clear (G3). VM live-migration (QEMU) and

   microVM (firecracker, infra-RFC-001 Phase 3) are separate, heavier seams.

3. ManagedCluster topology DSL (G5) — how a requester declares "3 nodes,

   anti-affinity across hosts, 1 LB" — likely a CRD schema (FLEET-011).

4. Hub/member trust — mTLS + Koder ID service accounts between tiers.
5. Tenancy axis at global scope — Fleet namespaces vs. the Stack

   Org→Workspace→Project model (RFC-017). Must reconcile before G7 UI.

7. Non-goals (for this RFC)

• Re-implementing the single-cluster primitives (done: FLEET-011..022).
• A bespoke storage engine — G1 reuses kdb (stack-RFC-001), no new substrate.
• Replacing koder-lease — Fleet consumes it, does not absorb it.

8. Sequencing

G1 (durable state) is the gate. Recommended order: G1 → FLEET-002 (multi-node) → G2 (global scheduler) → G6 (federation registry) → G3 (kbox migration) → G4 (rebalancer) → G5 (managed clusters) → G7 (universal UI/profiles). Each is its own epic; this RFC is the umbrella. The monitoring + control epic (FLEET-022..026) is orthogonal and proceeds in parallel — it makes each leaf observableactuatable, which G4G7 then consume globally.

9. Relationship to existing tickets

• FLEET-002 (multi-node + per-tenant namespace isolation) becomes the first

  concrete slice of G1+G2.

• BOX-142/143 (process stream + exec/signal) feed G4 (telemetry) and G7 (act).
• infra-RFC-001 (runtime strategy) + infra-RFC-003 (lease) are the

  runtime + lifecycle substrate this builds on.