Fleet — Container Orchestration Platform
- Area:
infra/net/ - Path:
infra/net/fleet - Slug:
koder-fleet(CLIkfleet) - Kind: Kubernetes-class container orchestrator (control plane + node agent)
- Version: v1.0.0 (
koder.toml [project]; README corrected from inflated v2.2.0 incce94b974f) - Self-hosted pair: Koder Fleet ⇄ Kubernetes — not yet registered in
registries/self-hosted-pairs.md; registration tracked as a follow-up (see Gaps).
Role in the stack
Fleet is the Koder analogue of Kubernetes: it schedules and supervises containerized workloads across a cluster, on top of the infra/net/kbox (kbox) runtime. Where Box runs a container on one node, Fleet decides which node, keeps the declared replica count alive, rolls updates, and wires service discovery / ingress.
Dual implementation (Go canonical, Koda port in progress)
| Impl | Path | Size | Tests | Status |
|---|---|---|---|---|
| Go reference | internal/ + cmd/ |
~9K LOC | 100+ (all green) | canonical — the binary that builds/ships today |
| Koda self-hosted port | src/ (.kmd) |
~22K LOC | 5 files | in progress toward the self-hosted flip (self-hosted-first.kmd) |
The two converged on the core object set (StatefulSet + DaemonSet in FLEET-015; ConfigMap + Secret in FLEET-016). Until the self-hosted flip, the Go reference defines what Fleet does.
Kubernetes parity matrix (canonical Go reference)
Audited 2026-05-31 (FLEET-014). parity = controller/type present and unit-tested; stub = declared as a config knob or struct field with no object/controller behind it; absent = no trace in internal/.
| Kubernetes primitive | Fleet (Go) | Evidence |
|---|---|---|
| Deployment | parity | internal/controller/deployment.go + tests |
| ReplicaSet | parity | internal/controller/replicaset.go + tests |
| Job / CronJob | parity | internal/workload/job.go + tests |
| Pod | parity | internal/workload/pod.go + tests |
| Service + DNS | parity | internal/network/{service_discovery,dns}.go + tests |
| Ingress | parity | internal/network/ingress.go + tests |
| HorizontalPodAutoscaler | parity | internal/autoscale/ + tests |
| Scheduler (affinitytaintstolerations) | parity | internal/scheduler/ + tests |
| NetworkPolicy | enforced | internal/workload/networkpolicy.go (kind: podSelector + ingressegress rules, nspodns-label peers + ports) + `internalnetworkpolicy.go pure EvaluateIngressEvaluateEgress (default-allow until selected, then default-deny) + store-backed PolicyEngine + API CRUD (FLEET-030). **Dataplane enforcement wired (FLEET-040):** PolicyEngine.ResolveVerdict expands selectors → concrete peer CIDRs (internalnetworkverdict.go); the node agent programs it per pod-container via BoxClient.ApplyNetworkPolicy → kbox POST containers{id}network-policy` → nftables in the netns (BOX-147, e2e-proven). FLEET-041 closed: proven live e2e on a real daemon container (after BOX-150151), strict fail-closed (config network_policy_strict, default off → flip after kbox-endpoint rollout), and re-apply-on-change (the agent reconcile loop re-resolves + re-applies a running pod's verdict when a policy or peer-pod IP changes). |
| PersistentVolume / PVC | parity | internal/storage/ + tests |
| Namespace + quotas | parity | internal/controller + internal/core; per-namespace ResourceQuota (internal/workload/resourcequota.go) enforced by admission.ResourceQuotaEnforcer (cpumemorypods caps) + API CRUD + tests (FLEET-002 axis B) |
| Health probes (livenessreadinessstartup) | parity | agent probe runner + tests |
| StatefulSet | parity | internal/controller/statefulset.go — stable ordinals, per-replica PVC reuse, OrderedReady/Parallel + tests (FLEET-015) |
| DaemonSet | parity | internal/controller/daemonset.go — one pinned pod per eligible node, selector/taint aware, node-leave reconcile + tests (FLEET-015) |
| ConfigMap | parity | internal/workload/configmap.go + API CRUD + env/volume projection (ResolvePodConfig) (FLEET-016) |
| Secret (object) | parity | internal/workload/secret.go + API CRUD + envvolume projection; sealed at rest via `internalsecret AES-256-GCM (Sealer seam; **DEK from a Koder KMS envelope** — secret.Resolve` unwraps a wrapped-DEK at boot via OAuth client_credentials, fail-closed; base64 key = dev fallback) (FLEET-016 + FLEET-021) |
| ServiceAccount | parity | internal/rbac SA objects + HMAC-signed projected tokens (MintServiceAccountToken) + default SAs seeded per namespace (FLEET-017) |
| RBAC (RoleClusterRoleBinding) | parity | internal/rbac RoleClusterRoleRoleBinding/ClusterRoleBinding + default-deny Authorizer enforced by API middleware; system:masters cluster-admin bootstrap (FLEET-017) |
| CNI | seam | internal/network/cni.go — NetworkDriver interface (SetupPod/TeardownPod + IPAM) + CNIRegistry selecting by cni_plugin; in-tree bridge driver (sequential CIDR IPAM, idempotent, derived MAC); 3rd-party drivers register at runtime. NetworkPolicy enforcement = #030. Behavioural pod-net wiring into the reconcile loop = follow-up (FLEET-018) |
| CSI | seam | internal/storage/csi.go — CSIDriver interface (ProvisionAttachMountUnmountDetachDelete) + CSIRegistry selecting by StorageClass; in-tree localhostpath drivers; 3rd-party drivers register at runtime. The existing StorageController (PV/PVC bind) is unchanged; behavioural wiring of the driver into provisioning = follow-up (FLEET-018) |
| LeaderElection / HA control plane | parity | internal/lease — coordination Lease + active-passive leader elector (acquirerenewfailover); control-manager runs the reconcile loop only while leading (FLEET-019) |
| CRD / custom controllers | parity | internal/crd — CRD registry + schema validation + dynamic /apis/{group}/{version}/… REST + controller-runtime-style reconcile loop (FLEET-011) |
| Watch API (subscribe) | parity | internal/api/watch.go — SSE GET /api/v1/watch/{kind} + namespaced route over core.Store.Watch; RBAC watch verb; disconnect cleanup (FLEET-022) |
| Admission webhooks | parity | internal/admission — chain (mutating→validating) after RBAC: built-in DefaultServiceAccount + NamespaceExists plugins + external validatingmutating webhooks (fail-closed default) on poddeployment/CR creates (FLEET-012) |
| PodDisruptionBudget | parity | internal/workload/pdb.go (minAvailablemaxUnavailable, int or %) + `internalcontrollerdisruption.go CanEvict evaluator + eviction subresource POST /pods{name}/eviction` (409 when a budget would be violated); status reconciled each tick + tests (FLEET-013) |
kubectl exec / process control |
parity | internal/api/exec.go — POST /pods/{name}/{exec,signal} (the audited SSH replacement) behind three gates: dedicated RBAC verbs execsignal (not update), an opt-in ProcessControl consent kind (off by default), and the admission webhook chain; every action emits an audit Event; delegates to kbox ExecSignalPID (BOX-143) (FLEET-026) |
| per-workload resource metrics | parity | internal/procmetrics — collector polls each enabled ProcessMonitor's cgroup readout (BOX-142) and pushes obs-SDK gauges (cpumemiopids + process-count-by-comm, bounded labels) to the observe stack; sync `GET /pods{name}processmetrics`; opt-in + off by default (FLEET-025). Per-PID CPUmem = BOX-148 |
| Federation (member clusters) | parity (registry) | internal/federation — MemberCluster registry: RegisterHeartbeatDeregister + health derived-on-read from heartbeat TTL + cross-cluster Aggregate(kind) fan-out (errors recorded-not-fatal, not-Ready skipped); SA-token trust (FLEET-020). API /api/v1/federation/{members,members/{name}/heartbeat,aggregate/{kind}} (FLEET-035 / G6). Active-active + cross-region + mTLS = FLEET-037 (gated on kdb-RFC-008 + real 2nd region) |
| ManagedCluster (clusters-as-a-resource) | parity | internal/workload/managedcluster.go kind + internal/controller/managedcluster.go — reconciles N owned pods, across-host anti-affinity (each replica pinned to a distinct node) + optional LoadBalancer Service; status tracks ready/distinct-nodes; namespaced CRUD API (FLEET-034 / G5) |
Headline: the full core surface is real, tested, and enforced — workload kinds (DeploymentReplicaSetStatefulSetDaemonSetJobCronJobPod), ServiceIngressHPA/ schedulerNetworkPolicyPV-PVC/probes, ConfigMap + Secret (sealed at rest), RBAC + ServiceAccount + Koder ID human-token auth, CRDs + custom controllers (the operator/extensibility surface — FLEET-011), and admission control (mutating + validating webhooks after RBAC — FLEET-012), and active-passive HA (coordination-lease leader election — FLEET-019). The remaining gaps are (a) pluggable storage/network driver interfaces (CSI/CNI — FLEET-018) and (b) PodDisruptionBudgets (FLEET-013) — the last plug-point + one drain-safety guard, not core primitives. The full k8s request pipeline (authn → authz → admission) is present, and the control plane is no longer a SPOF.
Surfaces
| Surface | Form | Notes |
|---|---|---|
koder-fleet server |
Go control plane | API server + scheduler + controller manager + DNS + ingress |
koder-fleet agent |
Go node agent | container runtime bridge (kbox via Unix socket), probe runner, resource monitor |
kfleet CLI |
Go binary | apply/get/delete/logs/exec/scale/rollout/nodes/namespaces + drain/drain-status/schedulable (FLEET-045) |
| node-update drain | REST subresource | POST/GET /nodes/{name}/drain + POST /nodes/{name}/schedulable — the kodix#079 planned-maintenance contract (FLEET-045) |
| Hub apt roles | .deb (kodix#077) |
koder-fleet-agent + koder-fleet + role metapackages kodix-node/kodix-fleet; opt-in, NOT in the base image; first-boot SVID enrollment (FLEET-044) |
| REST API | HTTP :9300 |
30+ endpoints; agent :9301, metrics :9302, DNS :5353, ingress :8080/:8443 |
/metrics |
Prometheus | built-in |
Primary couplings
| Component | Relationship | Status |
|---|---|---|
infra/net/kbox (kbox) |
container runtime fleet drives via Unix socket (BoxClient) |
live |
infra/net/lease (koder-lease) |
semantically adjacent (leaseTTL); Fleet's own `internallease LeaderElection is **distinct** (k8s coordination.Lease`-style leader election, NOT compute-TTL reaping). Consolidation evaluated under FLEET-014/019 → kept separate (different concept) |
distinct |
Gaps / open tickets
done/: 014 (audit), 015 (StatefulSet + DaemonSet), 016 (ConfigMap + Secret + at-rest sealing), 017 (real RBAC + ServiceAccount + API authz), 011 (CRDs + custom controllers), 012 (admission webhooks), 019 (LeaderElection/HA), 020 (Koder ID human-token JWKS + self-hosted pair), 022 (watch/subscribe SSE API), 024 (event→alert pipeline via koder-notify — internal/alert AlertRule CRD + controller + dedup), 013 (PodDisruptionBudgets), 018 (CSI/CNI driver seams), 030 (NetworkPolicy cross-namespace isolation — control-plane half: kind + pure eval engine + PolicyEngine + API; dataplane packet-drop = BOX-147), 002 (multi-node + per-tenant isolation — all three axes closed: A multi-node membership present/deep distributed runtime BOX-gated, B ResourceQuota done, C NetworkPolicy done as 030); monitoring/control epic COMPLETE — 023 (ProcessMonitor + internal/procmon events), 025 (per-workload resource telemetry — internal/procmetrics collector + obs gauges + sync endpoint; per-PID CPUmem = BOX-148), 026 (mediated execsignal SSH replacement — three gates + audit). backlog/pending/: the global control-plane campaign (infra-RFC-004) is LAUNCHED — anchor 027 decomposed into 031–038 (G2–G7 + G6b + G7-UI). DONE: 031