Backup & Disaster-Recovery Target Architecture — self-hosted-first object store + failure-domain tiering

draft

Status

draft — opened 2026-05-31. The Stack has no backup/DR architecture doc today (a grep of policies/ + specs/ finds none); per-service backups are configured ad-hoc (e.g. Flow's nightly kzip → Google Drive). This RFC fills that gap. Design only — no implementation.

T2 provisioned + validated 2026-05-31. The dedicated cloud bridge (T2) is now concrete and end-to-end tested — see §T2 provisioning below. The remaining T2 work (repoint the prod Flow nightly + first full backup) is gated on the flow#214 owner. T1 (self-hosted on kdrive) and T3 (2nd failure domain) remain design-only.

Summary

Where should Koder Stack backups live, long-term? Today the reference implementation (Koder Flow, FLOW-214) backs up to Google Drive via a personal OAuth user token (rpm32510@gmail.com). That choice has two structural problems:

  1. External dependency — Google Drive violates self-hosted-first.kmd; the

    Stack is concurrently building a self-hosted object store (kdrive/MinIO, kdb-obj per stack-RFC-006).

  2. Per-user quota contention — a Google per-user limit (~750 GB/day upload +

    API rate limits) means two heavy jobs on the same account (the backup + an active Drive migration) throttle each other. Observed live 2026-05-31: Flow's nightly stalled at 0 KiB/s for >1h30 in userRateLimitExceeded backoff.

But the naive fix (self-host the backup on kdrive) is incomplete: every Koder host today is in one failure domain (EVEO). A backup in the same DC as the primary is not off-site — it does not survive DC loss. Google Drive, for all its faults, IS a distinct failure domain.

This RFC ratifies a tiered target, not a single choice.

Decision — three tiers

Tier Target Failure domain Purpose Status
T1 — self-hosted primary kdrive (MinIO, S3-compat, live LXC) via restic/kzip S3 backend EVEO (same as primary) fast local restore; the self-hosted backup of record; foundation T3 extends buildable now (kdrive exists)
T2 — off-site bridge (cloud) Google Drive, but structured: dedicated service account + Shared Drive (org asset, own quota), NOT a person's My Drive Google (distinct) true off-site DR until a 2nd Koder failure domain exists bridge; self-hosted debt
T3 — self-hosted off-site (north star) kdrive→kdrive (or kdb-obj) replicated to a 2nd Koder failure domain a 2nd Koder DC/site sovereign off-site DR; retires T2 (Google) gated on infra that does not exist yet

This is 3-2-1 done right: ≥2 media (T1 self-hosted + T2/T3 off-site), ≥1 off-site (T2 now → T3 later), and the off-site is in a different failure domain.

Why not the alternatives (Quality > Speed, stack-principles §2)

  • Reuse the DWD service account impersonating a person's Drive (rejected).

    Solves quota fastest but couples prod backups to a human account (rodrigo@vivver.com.br), no clean revocable identity, deepens Google lock-in silently. A "v1 with limitations" the §2 ruler forbids when a cleaner form (T2's dedicated SA + Shared Drive) is reachable at small extra cost.

  • Self-host on kdrive and call it done (rejected). Same failure domain (EVEO)

    → not off-site → not DR. Necessary (T1) but not sufficient.

  • Stay on Google Drive personal account (rejected). The status quo; the quota

    contention + lock-in are exactly what this RFC corrects.

Trade-off accepted: more work now (stand up T1 on kdrive + structure T2 + open the 2nd-failure-domain effort) instead of the quick SA patch — bought to avoid deepening external lock-in and to put the path to sovereignty (T3) on the board.

The real blocker is not the credential — it's the failure domain

Full self-hosted off-site (T3) is gated on a second Koder failure domain (a 2nd DC or site). Today all hosts (s.khost1, s.r1, s.ops*) are EVEO. Standing up a 2nd domain is its own infra effort (a follow-up RFC under infra/net); until it lands, T2 (structured cloud) is the legitimate off-site tier. This RFC's most important finding is to name that gap explicitly rather than pretend kdrive-in-EVEO is DR.

Self-hosted debt

Added to meta/docs/stack/registries/self-hosted-pairs.md (row 20, 2026-05-31): the pair koder backup target × google_drive, state shadow/bridge, flip target = kdrive/kdb-obj cross-failure-domain (T3), gate = a 2nd Koder failure domain exists.

T2 provisioning state (2026-05-31)

The dedicated-SA + Shared-Drive form of T2 (the §Decision "off-site bridge") was stood up and validated this session — replacing the rejected "personal My Drive via rpm32510 OAuth token" status quo that suffered the per-user quota contention:

Artifact Value
Service account koder-flow-backup@koder-dek.iam.gserviceaccount.com (project koder-dek, created in Console by owner; no project IAM roles — Drive-only)
JSON key meta/context/credentials/koder-flow-backup-sa.json (gitignored plaintext) + .json.enc committed (vault scheme: aes-256-cbc -pbkdf2 -iter 200000)
Shared Drive Koder Flow Backups, id 0AOyPgucC0Mp9Uk9PVA (owner org root@root.ooo Workspace — has org pooled quota, not a person's My Drive)
SA membership fileOrganizer (Content manager) on the Shared Drive — least-privilege: full content ops (createwritedelete for restic prune), no member management
rclone form :drive,service_account_file=<sa.json>,team_drive=0AOyPgucC0Mp9Uk9PVA,scope=drive:

Why the quota problem is gone: the SA's Drive API calls bill to the koder-dek project's own quota bucket, decoupled from both the rpm32510 per-user upload limit (the original stall) and rclone's shared-client per-minute limit. The Shared Drive's storage is the root.ooo Workspace org pool (45 TiB total observed), not a personal 15 GB / 750 GB-day account.

Validation performed (this session, from the laptop, then torn down):

  1. rclone via the SA: lsd / mkdir / write / read-back (content match) /

    purge against the Shared Drive — all clean, no quota error.

  2. Full restic stack through the SA remote: init → backup (snapshot

    ca45255b) → check ("no errors were found") → restore → byte-identical diff → repo purged. Proves restic→rclone→SA→Shared Drive end to end.

Still gated (flow#214 owner — prod, locked by another session):

  • Push the SA key into the flow LXC (/etc/koder-flow/) + add the

    gdrive-sa rclone remote there (additive; coordinate with the active flow#214 backup session).

  • Repoint koder-flow-backup.service from rclone:gdrive: (rpm32510) to

    rclone:gdrive-sa:koder-flow-kzip.

  • Kick the first full backup to T2 (the ~178 GiB initial; nightly is

    incremental thereafter).

Implementation tickets (described, NOT executed — gated)

  • flow#216 (proposed, productsdevflow/engine): add a T1 restic/kzip→kdrive

    (S3) backup of Flow, parallel to the Drive one; flip the Drive job to T2 (dedicated SA + Shared Drive). Owner: flow#214 session — prod backup config.

  • GCP (human step): DONE 2026-05-31. Dedicated backup SA koder-flow-backup

    created in koder-dek (Console, owner) + Koder Flow Backups Shared Drive (0AOyPgucC0Mp9Uk9PVA) + SA added as Content manager + key .enc committed. See §T2 provisioning.

  • infra/net (proposed RFC): establish a 2nd Koder failure domain (2nd DC /

    site) — the prerequisite for T3. The single biggest item.

  • kzip/restic S3 backend validation against kdrive MinIO.

Out of scope

  • The Flow-specific backup mechanics (FLOW-214 owns those); this RFC sets the

    target architecture every Koder service backup should converge on.

  • Backup alerting/silencing — see FLOW-215 + infra-RFC-005 (maintenance-window

    silences).

Cross-references

  • stack-RFC-006 (object-storage-plane — the self-hosted target substrate)
  • self-hosted-first.kmd (the principle this serves)
  • FLOW-214 (current cloud backup architecture — becomes T2)
  • specskzipformat.kmd (the backup format)
  • infra-RFC-005 (maintenance-window control plane — alert silences for backup runs)