Maintenance-Window Control Plane — declarative change windows reconciled by Fleet

Status

draft — opened 2026-05-31 from a live operational gap (a stalled Flow backup that should have alerted but didn't, and an upcoming CIMMG maintenance window with no coordination mechanism). Design only; not yet scoped into implementation tickets.

Summary

Today a planned change / maintenance window in the Koder Stack has no single declarative owner. The four audiences that must react to "service X is down / changing from T0 to T1" are coordinated by four unrelated mechanisms:

This RFC proposes modeling a maintenance window as a declarative resource reconciled by the Stack's existing global control plane (Koder Fleet, infra-RFC-004) — NOT a new bespoke orchestrator — fanning out to the existing edges. One source of truth, auditable, GitOps-able.

Motivation

Two concrete incidents on 2026-05-31 exposed the gap:

1. Stalled prod backup, no alert. koder-flow-backup.service (the Flow nightly

   kzip → Drive) hung for >1h30 in Google Drive per-user rate-limit backoff (0 KiB/s, both kzip+rclone idle). The backup-failure notifier (koder-flow-backup-alert.service, FLOW-215) was inactive/dead, so nothing surfaced it. Conversely, had it been wired, a planned migration window would have paged falsely — there is no way to say "this downtime is expected, suppress."

2. CIMMG maintenance window (infra-RFC-001 Phase 0/2). Migrating production

   healthcare LXCs needs a coordinated window: warn AI sessions off flow/CIMMG components, show end users a maintenance page, suppress alerts, pause backups. Each is hand-coordinated today.

As autonomous operation scales (dozens of concurrent AI sessions + a growing service fleet), ad-hoc coordination develops seams: false pages, missed suppressions, deploy-into-downtime races, manual maintenance pages forgotten.

Decision

Model the maintenance window as a Fleet CRD + controller (reuse-first), fanning out to existing edges. Do NOT build a parallel maintenance orchestrator.

infra-RFC-004 already ratifies Koder Fleet (infra/net/fleet) as the Stack's declarative global control plane — it has crd/, controller/, admission/, alert/ subsystems. A maintenance window is just another declarative resource it reconciles. Building a separate maintenance daemon would duplicate Fleet's reconciliation loop and koder-notify's alerting — the exact "parallel control plane / duplicated plumbing" anti-pattern rejected in infra-RFC-001 §Axis C and governed by reuse-first.kmd.

The resource

apiVersion: fleet.koder.dev/v1
kind: MaintenanceWindow
metadata: { name: cimmg-phase0-cutover }
spec:
  start: 2026-06-02T02:00:00Z
  duration: 4h
  reason: "infra-RFC-001 Phase 0 — Docker → kbox cutover (CIMMG PoC)"
  targets:
    services:  [koder-flow]              # logical services
    vhosts:    [flow.koder.dev]          # Jet edge
    components: [products/dev/flow, infra/net/box]  # AI-session scope
    pauseTimers: [koder-flow-backup.service]
  silenceAlerts: true                    # koder-notify silence for the targets

The controller fan-out (reconcile on window active/expired)

On window expiry the controller reverses every effect (un-silence, un-maintenance, resume timers, retract notice). Idempotent, reconciled — a missed tick self-heals.

Why not the alternatives

• Standalone maintenance daemon (rejected). Duplicates Fleet's reconciliation +

  notify's silencing → reuse-first.kmd violation; a second control plane to operate. Same trap avoided in infra-RFC-001 (kbox not a god-object).

• Status quo (rejected long-term). Works at today's scale but the seams above

  don't survive growing autonomous ops; not auditable, not declarative.

• Put it all in Jet (rejected). Jet is the HTTP data-plane edge (reverse

  proxy/TLS), not a control plane — it serves the maintenance page but must not own job pausing, alert silencing, or session coordination (web-server.kmd scopes Jet to web-serving).

Trade-off (Quality > Speed, stack-principles §2): extend the existing control plane (Fleet CRD) + add a silence primitive to notify, rather than a new orchestrator — more correct long-term (reuse-first + declarative/reconciled) at the cost of depending on Fleet's CRD/controller foundation maturing.

Phasing

The CRD/controller waits on Fleet's foundation (mid-build per infra-RFC-004 + active FLEET tickets). The first two slices are independently useful and become the controller's reconciliation targets later:

Implementation status (2026-06-20): Phase 3 started under FLEET-052 slice 5 — the MaintenanceWindow kind + controller (Pending→Active→Expired) landed in infra/net/fleet, with the first reconciliation effect: a window freezes ServiceDeployment rollouts to its targeted services/namespaces while active. The remaining fan-out (Phase 1 notify-silence, Phase 2 Jet maintenance flag, Phase 4 notice emission) is not yet wired. This RFC stays draft until the owner ratifies; the implemented subset is additive and reversible.

Out of scope

• Replacing the koder-lock/notice convention for ad-hoc per-component work locks

  — those stay (they're about who's editing what, not planned downtime). This RFC only adds declared maintenance windows as a controller-managed overlay.

• Approval workflows / change-management bureaucracy. A window is a fact to fan out,

  not a ticket to approve.

Open questions

• Q1 — silence model in notify: label-matcher (Alertmanager-style) vs

  service-id list. Lean matcher for composability.

• Q2 — timer pause mechanism: controller systemctl over SSH/incus vs a

  per-unit ConditionPathExists=!/run/koder-maintenance/<svc> drop-in (no remote exec; the controller just touches a file). Lean the drop-in (declarative, no SSH).

• Q3 — does the window auto-pause AI deploys, or only warn? Warn first (notice);

  a hard gate (block deploy hooks during a window) is a later tightening.

Cross-references

• infra-RFC-004 (Fleet control plane — the home for the CRD)
• infra-RFC-001 §Phasing (CIMMG window is the first real consumer)
• FLOW-215 (backup-failure alerting — the silence's first beneficiary)
• policies/observability-first.kmd (silences are an observability primitive)