Shadow / Parallel-Run Validation — validate risky backend changes in prod without blocking windows

Document format reference: meta/docs/stack/policies/document-format.kmd

Status: RATIFIED 2026-06-06 (owner). The §10 decision points are accepted as recommended: (1) two-tier model (generic SDK shadow-compare harness + kdb-native WAL-ridden shadow); (2) name koder_shadow_kit / "Koder Shadow" (registry row lands with the first koder.toml, per naming-aliases.kmd); (3) shadow becomes the default and the gated window the exception (Phase 3 updates always-on); (4) the side-effect-free boundary (I3) is accepted — side-effecting changes keep flag-gated rollout; (5) promotion thresholds (N/T) are per-consumer config with a Stack default. Implementation tracked in engines/sdk/koder_shadow_kit/backlog/ (Phase 1, #001–#004) + cross-component Phase 2 (kdb, RFC-008 #776 reuse) + Phase 3 (always-on policy update).

Phase 2 (kdb) — DONE 2026-06-22 (kdb#784, shadow#005). The kdb-gateway is the first productionized Tier-2 consumer: default-off --shadow-backend flags open a second store, replay the gateway's own WAL into it (reusing #776 WalApplier), compare vs the authoritative store, and emit the SSOT contract shadow_divergence_total{component,divergence_kind} + shadow_reads_total{component} on /metrics. Homologated end-to-end on real binaries (3-arm cross-process e2e + 387/0 lib tests).

Phase 3 (always-on flip) — DONE 2026-06-22. always-on.kmd R3.5 now makes the shadow gate the DEFAULT parity validation for data-risky storage/transform changes (continuous, non-blocking), with the gated test-window retired to the EXCEPTION — reserved only for genuinely un-shadowable changes with external side effects (the I3 boundary). All three phases shipped; RFC-011 is fully realized (per-consumer promotion thresholds N/T per §10.5 remain a config detail).

Origin: owner proposal (2026-06-06) — instead of gated test windows that serialize work across the Stack (block other components' tickets while a risky change is validated in prod), a reusable primitive so a new backend feature with data-loss potential runs in shadow against production traffic: the old path stays authoritative and intact while the new path is written/computed in parallel and compared, never trusted until parity is proven. Validation becomes a continuous, non-blocking background process instead of a scheduled event.

1. Context & problem

The Stack's current way to validate a risky, potentially-irreversible backend change (e.g. a kdb storage-engine change that could corrupt or lose data) is a gated test/migration window: an owner schedules a coordinated moment, and — because the change is destructive and shares infra — other work on that component (and often adjacent ones) blocks until the window closes. Two live examples from this very campaign:

• RFC-017 / id#196 CONTRACT — dropping legacy tenancy columns is gated on an

  owner window + decisions A/B, because nothing automatically proves the new axis reads are byte-for-byte equivalent to the old ones. So the contract sits blocked.

• RFC-018 SSO prod enablement — the flag flip is owner-gated partly because

  the new path's behaviour in prod is only knowable by turning it on.

This window-tax is in direct tension with three ratified principles: always-on (no destructive migration without a checkpoint), hyperscale-first, and the current acceleration phase goal of *not blocking parallel work across components*. The window is a coordination bottleneck, not a technical necessity.

The well-known industry answer is shadow traffic / parallel-run / dark-launch with comparison (GitHub's "Scientist", Google's shadow reads, Stripe's dual-writes). The Stack already practices the manual halves of it (see §3); it lacks a turnkey, reusable primitive.

2. Goals / non-goals

Goals

• A reusable Stack primitive that lets a new backend implementation run *n

  shadow of the authoritative one in prod, with automatic divergence detection and telemetry-driven promotion*— zero data-loss risk, zero blocking of other components' work.

• Make "validate in prod" continuous and parallel, retiring the gated window

  as the default mechanism for data-risky changes.

• Cover both (a) logic-over-a-store changes (any component) and (b)

  storage-engine changes (kdb specifically).

Non-goals

• Changes with external side effects (send email, charge, mutate a third

  party) — shadowing those is unsafe; they keep flag-gated rollout (§7.4).

• Replacing always-on's expand→migrate→contract — this operationalizes it

  with an automated comparator, it does not supersede it.

• A general A/B experimentation framework for product features (this is

  correctness-parity validation, not metric optimization).

3. Prior art in the Stack (what exists, what's missing)

Conclusion: the ingredients exist; this RFC unifies them into one primitive.

4. Design — two tiers

4.1 Tier 1 — generic shadow-compare harness (SDK)

A koder_*_kit-style module (proposed name §8) wrapping any store/RPC interface:

• write: route to the authoritative (old) path AND, best-effort, to the

  shadow (new) path.

• read: serve from authoritative; sampled + asynchronous, also read from

  shadow and compare (semantic, §7.2).

• emit each divergence as an observability signal (infra/observe/,

  correlated by trace_id, with a bounded-cardinality divergence_kind label per observability-first.kmd), plus a rolling promotion verdict (§6).

• For components whose risky change is in logic over a store (id, talk, chat…).

Language bindings: Go first (most backends), then the other SDK languages as consumers appear (reuse-first — bind, don't fork).

4.2 Tier 2 — kdb-native shadow via the WAL (the Koder-specific win)

For changes to kdb's own storage engine/format, do not dual-write at the client (it doubles the hot path and creates two-store atomicity problems, §7.1). Instead ride the WAL, which already records every logical mutation:

• a shadow applier (reusing the RFC-008 WalApplier, #776 — the same code

  that applies WAL to replicas) materializes the new storage format from the same WAL into a side store;

• a comparator replays representative reads against both the old engine and the

  shadow store and reports divergence;

• the authoritative write path is untouched (one write, to the WAL/old

  engine), so there is zero hot-path cost and zero data-loss risk — the shadow is strictly downstream of the authoritative WAL.

Promote when divergence = 0 over the window → flip authority old→new → contract (drop old) per always-on R3.1.

5. Safety invariants (normative)

• I1 — Authoritative independence. The authoritative path MUST NOT depend on

  the shadow path. A shadow writereadcompare failure is logged, never surfaced to the user, never blocks or fails the authoritative operation.

• I2 — Shadow is never trusted. Shadow output is compared, not served, until

  the promotion gate (§6) passes.

• I3 — Side-effect-free only. Only pure storage/transform paths may be

  shadowed (§7.4). Paths with external side effects are out of scope and keep flag-gated rollout.

• I4 — Bounded window + reconciliation. Shadow windows are time/sample

  bounded; long windows require a reconciliation/backfill to counter drift (§7.5).

• I5 — Semantic comparison. Comparison is canonicalized/semantic, not

  byte-equality (§7.2).

• I6 — Authority flip is explicit + reversible. Promotion flips authority via

  config/flag (instantly reversible to the old path) before the irreversible contract step; contract follows always-on R3.1 only after a clean window.

6. Promotion gate (telemetry-driven)

Promotion is a measured decision, not a judgment call — it mirrors the G1/G2 gates of self-hosted-first.kmd:

• G-parity (functional): divergence rate = 0 over N samples AND ≥T elapsed

  on representative prod traffic (N, T per-consumer config).

• G-regression: no new error/latency regression on the authoritative path

  attributable to the shadow harness (it must stay off the hot path).

• Optional G-perf (for self-hosted-first replacements): the new path is ≥ as

  fast as the old before flip.

A consumer that cannot reach 0 divergence does not promote — the harness keeps running (non-blocking) while the new path is fixed.

7. Sharp edges & mitigations

1. Dual-write atomicity (Tier 1). Two stores aren't atomic; old-ok/new-fail

   is a write-path artifact, not a logic bug → I1 (old authoritative, new best-effort). Tier 2 sidesteps this entirely (single WAL write).

2. Semantic comparison. Timestamps, ordering, generated IDs diverge

   legitimately. Need canonicalization + ignore-hooks (we already hit this in

   776: Insert.row_key is a debug string, not a key → compare decoded rows).

3. Cost. Shadow reads MUST be sampled + off-hot-path (async) so they never add

   user-facing latency.

4. Side effects. kdb storage = perfect fit; "emit invoice / send email" =

   excluded (I3).

5. Long-window drift. Out-of-band writes, GC, TTL drift the two stores →

   bounded window + reconciliation (I4).

6. Comparator coverage. Read-compare only catches divergence on paths actually

   exercised; pair with a synthetic replay corpus for cold paths.

8. Naming (owner decision)

Proposed: koder_shadow_kit (Tier-1 SDK module, koder_*_kit convention) + the concept name "Koder Shadow". Per naming-aliases.kmd / registries/component-names.md, the final name is owner-ratified and its registry row lands with the first koder.toml. Alternatives for the owner to weigh: a distinctive codename (cf. KomposeKaviIRIS) vs the descriptive shadow.

9. Rollout (phased, non-blocking)

• Phase 1 — Tier-1 Go harness in engines/sdk/ + the observability divergence

  signal + the promotion-gate evaluator. Dogfood on a low-risk store wrapper.

• Phase 2 — Tier-2 kdb WAL-ridden shadow applier + comparator (reuses #776).

  First real target: a kdb storage-format change behind shadow instead of a window.

• Phase 3 (DONE 2026-06-22) — retired "gated window" as the default for

  data-risky changes; always-on.kmd R3.5 points CONTRACT-class storage/transform work at the shadow gate. Windows remain only for genuinely un-shadowable changes (external side effects, the I3 boundary).

• Each phase ships independently and blocks nothing (the harness is additive +

  flag-gated).

10. Owner decision points (ratification gate)

1. Adopt the two-tier model (generic SDK + kdb-WAL-native) vs SDK-only. (Reco:

   two-tier — the WAL tier is where the biggest data-loss risk lives and it's the cheapest/safest integration.)

2. Name (§8).
3. Default-flip of the window policy (Phase 3): make shadow the default and

   windows the exception — confirm this is the desired end-state for always-on.

4. Scope of "side-effect-free" — confirm the I3 boundary (storage/transform

   only) is acceptable, i.e. side-effecting changes stay window/flag-gated.

5. Promotion thresholds — who sets N/T per consumer (per-component config vs a

   Stack default).

11. Open questions

• Cross-language parity: Tier-1 is Go-first; kdb is Rust (Tier-2 lives there

  natively). Do non-Go/Rust consumers need a binding soon, or is Tier-2 + Go enough for the foreseeable backend risk surface?

• Comparator corpus: is prod-traffic sampling sufficient, or do we standardize a

  synthetic replay corpus per store for cold-path coverage?

• Relationship to koder_test_* doubles (clockrngstate): can the shadow

  comparator reuse the determinism kit to canonicalize non-deterministic fields?