kdb-next schema evolution runbook

When you need this. A Koder service whose repository is backed by kdb-next (pkg/kdbnext, gRPC EnsureTable) fails to boot after a deploy because the table schema in the running master binary no longer matches the schema the kdb-next gateway already holds for that tenant. This runbook is the decision procedure + the mechanics for every path out of that wall.

The kdb-next gRPC EnsureTable contract is strict: any mismatch between the schema a binary declares and the schema the catalog already holds is a hard boot error. (The legacy kdb-1 HTTP path tolerated additive column-bag changes silently — kdb-next does not.) That strictness is correct (it surfaces drift instead of silently corrupting reads), but it means every EnsureTable change on a kdb-next-backed table is a schema migration and must be planned, not shipped by accident.

This is the kdb-next companion to the Postgres story in always-on.kmd §3 (zero-downtime schema evolution) and the generic expand→migrate→contract recipe in specs/migrations/expand-migrate-contract.kmd.

1. When the bootstrap fails

Symptom

A freshly-built binary crash-loops at startup with a kdb-next bootstrap error naming the table and the word schema:

ERROR kdb-next admin stores bootstrap failed
  error="audit: ensure admin_audit_log: kdbnext: EnsureTable(\"koder-id\"/\"admin_audit_log\"):
        rpc error: code = Aborted desc = table tenant:.../\"admin_audit_log\"
        already exists with a different schema"

The currently-running prod binary keeps working — it compiles against the schema the catalog still holds. Only the new binary trips the check, so the immediate mitigation is always: roll back to the last-known-good binary (keep a *.bak-pre-<ticket> copy on every deploy) and resolve the drift out-of-band. Prod is never down while you decide.

Diagnose — read both sides

1. Catalog side (what the gateway holds today):

`bash export KDB_ENDPOINT=https:/gateway-host>:18090 export KDBAUTHTOKEN="$(cat etckoder-idkdb-nextbearer-token)" export KDBTLSCA=etckoder-idkdb-nextca.pem # only for https endpoints

kdbctl inspect tenant tenant table table

`

2. Master side (what the new binary declares): read the EnsureTable

   call in the repository's *_kdbnext.go — column list, types, primary key, indexes. The diff between the two is the migration you must perform.

Classify the diff — it picks the path in §2:

2. Decision matrix

There are three sanctioned paths. Pick by how coherent the history must stay, not by what is fastest.

a) Catalog rename to *_v2 — fast, zero-downtime, splits history

The binary bumps the table name constant (admin_audit_log → admin_audit_log_v2); kdb-next creates a fresh table with the new schema. The old table stays in the tenant as a read-only archive (rows still recoverable by direct PK lookup).

• Wire-compatible under always-on.kmd §2: kdb-next stores opaque proto

  payloads, so the row shape inside the gateway is unchanged — only the catalog key differs. Old readers/writers of the original table keep working; the new binary reads/writes _v2.

• Cost: history splits at the cutover instant. Queries that must span

  the boundary have to union both tables.

• Use for: append-only logs / event tables / disposable caches where the

  split is acceptable, or as an emergency unblock — but then track a follow-up StampRows/backfill ticket to reunify, and a DROP ticket for the archive once it ages out (per always-on.kmd §3 R3.3).

• Never use for: a primary entity table (users, clients, sessions)

  where queries must see one coherent history. Splitting those is data loss by another name.

There is also a first-class kdbctl migrate rename-table (updates the catalog name without row rewrites). The table fingerprint changes because the name is part of the canonical bytes, so follow a rename with migrate apply --kind stamp-rows to clear drift warnings on existing rows (or accept them on a soon-to-be-archived table).

b) kdbctl migrate apply --kind schema-only — proper additive migration

Installs the new schema in the catalog in place, O(1), preserving the table identity and all rows. This is the architecturally-right path for additive changes (new nullable column, new index) on a table whose history must stay coherent.

• Use for: primary tables, additive evolution, history must stay intact.
• Does not rewrite existing rows — old rows are read back against the new

  schema with the new (nullable) columns reading as null.

• For a non-null column, this is still a 3-step expand→backfill→contract

  (see §3 R3.2 of always-on.kmd): add nullable → backfill async → only then declare non-null in a later release.

c) kdbctl migrate apply --kind stamp-rows — reshape, rewrites rows

Installs the new schema and rewrites every row's fingerprint header, O(n). Needed when a column changes shape (rename, type change) so that existing rows carry the new canonical bytes.

• Use for: column rename / type change on a table that must keep one

  identity and history.

• Heavier: rewrites every row — treat as an online migration (run it as a

  tracked job; poll with migrate status / migrate get).

Tie-break (Quality > Speed, stack-principles.kmd §2). When the table carries real entity history, prefer (b)/(c) — the migration that keeps one coherent table — over (a) the rename, even though the rename is faster. The rename is reserved for append-only/disposable tables or a tracked emergency unblock.

3. Mechanics

Environment (all kdbctl migrate commands)

kdbctl reads connection config from global flags or env vars:

export KDB_ENDPOINT=https://<gateway-host>:18090
export KDB_AUTH_TOKEN="$(cat /etc/koder-id/kdb-next/bearer-token)"
export KDB_TLS_CA=/etc/koder-id/kdb-next/ca.pem

⚠️ Known blocker (as of 2026-05-29). kdbctl panics at startup on any https endpoint because the binary never installs a rustls CryptoProvider (rustls::crypto::ring::default_provider().install_default()) — the kdb-pgwire crate installs it at every TLS site, kdb-cli misses it entirely. Until that fix ships in infra/data/kdb, the plan/apply paths against the prod TLS gateway are not runnable; only path (a) (the name-constant rename, which needs no kdbctl) and migrations against a plaintext dev gateway work. Tracked as a kdb-cli bug; this runbook's migrate paths become operational the moment it lands. Build kdbctl on s.khost1.dev-linux-kdb (cargo build --release -p kdb-cli, ~1m).

Schema file (JSON)

migrate plan|apply take a --schema-file — a transcript of the protobuf Schema message. ColumnFile.ty is one of text, int64, bool, bytes, float64, timestamp.

{
  "name": "admin_audit_log",
  "primary_key": ["tenant_id", "id"],
  "columns": [
    { "name": "tenant_id",  "ty": "int64",     "nullable": false },
    { "name": "id",         "ty": "text",      "nullable": false },
    { "name": "actor_id",   "ty": "text",      "nullable": false },
    { "name": "action",     "ty": "text",      "nullable": false },
    { "name": "created_at", "ty": "timestamp", "nullable": false },
    { "name": "ua_carrier", "ty": "text",      "nullable": true  }
  ],
  "indexes": [
    { "name": "by_actor", "id": 1, "columns": ["actor_id", "created_at"], "unique": false }
  ]
}

Plan → Apply flow

# 1. Dry-run: show what would change, no writes.
kdbctl migrate plan  --tenant <tenant> --table <table> --schema-file ./schema.json

# 2. Apply. --kind schema-only (additive, O(1)) or stamp-rows (reshape, O(n)).
kdbctl migrate apply --tenant <tenant> --table <table> --schema-file ./schema.json \
       --kind schema-only

# 3. Track (StampRows returns a migration id; poll it).
kdbctl migrate status --tenant <tenant>            # most-recent-first list
kdbctl migrate get    --id <ulid>                  # one migration's state row
kdbctl migrate cancel --id <ulid>                  # cancel a pending/running one

# Adjuncts:
kdbctl migrate rename-table --tenant <tenant> --table <old> --new-name <new>
kdbctl migrate drop-index   --tenant <tenant> --table <table> --index-name <name>

4. Worked examples

Example A — admin_audit_log reshape (real, engine#186)

A master admin binary added 7 UA-carrier columns to the kdb-next admin_audit_log (append-only audit trail). The new EnsureTable (15 columns) collided with the deployed 8-column schema → boot crash.

• Class: whole-table reshape on an append-only table.
• Path taken: (a) catalog rename — bumped the constant to

  admin_audit_log_v2; kdb-next created the fresh 15-column table; the original stayed as read-only archive. History splits at 2026-05-28 (older admin events in the original, newer in _v2) — acceptable for an append-only audit log.

• Why not (b)/(c): the proper migrate apply path was blocked by the

  kdbctl rustls bug against the prod TLS gateway (§3 blocker). The rename needs no kdbctl, so it was the correct emergency unblock.

• Follow-up owed: drop the v1 archive via kdbctl once it ages past the

  retention window (after the rustls fix ships).

Example B — users gains a nullable locale column (hypothetical)

A primary entity table must keep one coherent history — the rename path is forbidden here.

• Class: additive, nullable.
• Path: (b) SchemaOnly. Author schema.json with the existing

  columns + { "name": "locale", "ty": "text", "nullable": true }, migrate plan to confirm the diff is additive, then migrate apply --kind schema-only. O(1); existing rows read locale as null until backfilled. If locale must become non-null, that is a later release (expand→backfill→contract, always-on.kmd §3 R3.2) — never a direct non-null add.

Example C — sessions.expires_at changes text → timestamp (hypothetical)

A type change on an existing column on a table that must keep its identity.

• Class: column type change.
• Path: (c) StampRows. Author schema.json with expires_at retyped

  to timestamp, migrate plan, then migrate apply --kind stamp-rows (O(n) — rewrites every row's fingerprint header). Poll migrate status / migrate get --id <ulid> to confirm completion before deploying the binary that reads the new type. Because old and new fingerprints differ, do not deploy the reader binary until the stamp completes, or stage it behind a read-both-shapes window (always-on.kmd §3 R3.1 expand→migrate→contract).

5. Gate

Every engine ticket that adds, removes, renames, or retypes a column (or adds/drops an index) on a kdb-next-backed table MUST reference this runbook in its acceptance criteria and name the path it takes (a / b / c). The migration mechanism is chosen explicitly, never by accident — an unplanned EnsureTable change is a deploy-time crash waiting to happen.

Cross-linked from always-on.kmd §3 (zero-downtime schema evolution) for the kdb-next case.