object storage plane and physical trust tiering

Status: Draft / exploratory. This RFC records an architecture conversation (2026-05-24) that started from "if git didn't exist, how would we design it today" and converged on a single structural principle that already runs, unannounced, across Flow, kdrive and the Hub. Nothing here is ratified. §8 lists the precise edits to normative artifacts that the owner would apply to ratify.

1. Thesis — the bytesmetadata (modelsubstrate) fronteira

There is one boundary that keeps reappearing across unrelated Koder components, and naming it explicitly clarifies a class of decisions:

Durable state splits into two planes with incompatible physics:

• Metadata / coordination plane — small, mutable, transactional,

  latency-sensitive, wants consensus (Raft/quorum), MVCC, a WAL and a buffer pool. This is kdb's substrate.

• Object / bytes plane — large, immutable, content-addressed,

  throughput-bound, wants erasure coding, no consensus on the bytes, streamingrange IO, and ref-counted GC. This is an object store.

The trap this RFC exists to prevent: treating "kdb is multi-model" as license to put bytes on the transactional substrate.

Multi-model ≠ multi-substrate. kdb is multi-model in the sense that relational / KV / time-series / graph / vector are *projections over one shared substrate* — the transactional KV/page store. Evidence: every kdb layer is explicitly derived from that substrate (RFC-001: kdb-ts = "hypertable-style sobre TiKV ranges", kdb-vec = "ANN sobre TiKV-stored embeddings", kdb-search = "derivado via CDC", kdb-stream = "Raft log → fan-out"). None of the existing models needed a different substrate.

Object storage is the first model that does. An "object storage model" is trivial as an API surface (PUT/GET by key — kdb-kv already does that), but its identity lives in the substrate physics, not the API: erasure coding, no-WAL on the bytes, large sequential extents, streaming, immutable ref-counted GC. Therefore:

Litmus test: does the new model reuse kdb-wal + kdb-buffer + MVCC + replication? If yes, it is a real kdb model. If it must bypass them, it is a new substrate wearing a model's clothes — and that is fine, as long as we call it what it is.

Precedent both ways: Ceph unifies object+block+file on one engine — but only because its substrate (RADOS) is object-shaped, and SQL ACID would be the misfit there. kdb chose a transactional substrate, so relationalgraphvector are natural and blob is the misfit. No single substrate is optimal for both. The honest end-state is two substrates under one brand, not one substrate for everything.

2. Current state (evidence, 2026-05-24)

This fronteira is already implemented across the Stack — it was just never named:

• stack-RFC-001 §1 (Non-goals) makes blob storage a *ermanent

  exception* bytes > 1 MB live in kdrive (MinIO / S3-compat). Stated reasons: TiKV not optimized for blobs; Raft replication of a 100 MB blob saturates I/O for no gain; MVCC GC of large blobs is complex. Coordination pattern: "kdb armazena metadata + ACL + signed-URL config; kdrive serve bytes" — the S3 + Postgres pattern.

• kdb has no blob layer, by design. RFC-001's planned layers are

  kdb-kv / kdb-ts / kdb-vec / kdb-search / kdb-stream; the 22 crates are substrate (kdb-kv-trait, kdb-buffer, kdb-wal, kdb-adapter), engine, surface, and domain (kdb-timeseries, kdb-graph, kdb-vector). None for object/blob.

• self-hosted-pairs.md has no MinIO entry. kdb's pair targets

  PostgreSQLSQLiteRedis/MySQL. MinIO (the object store under kdrive) is an external dependency with no Koder-native replacement tracked — a real gap relative to policies/self-hosted-first.kmd.

• Koder Hub is the live precedent. products/dev/hub/depot uses

  kdb-next as primary datastore for the registry/metadata plane — catalog/tenants, CRDT store, federation, attestations, signing keys, developer auth, artifact records (repo/schema.go AppArtifactsTable). The package bytes go through a separate Storage interface (depot/storage/storage.go) whose only implemented backend today is local filesystem (depot/storage/local.go, PackagesDir). Code comments show the abstraction was designed to front an object store via signed-URL/302 ("302 to the underlying object-store URL", handlers/apps_download.go:44; "stored in object store", services/delta_service.go:8). The object-store backend slot exists and is empty.

• Koder Flow stores git repos on the filesystem. Flow is a Forgejo

  fork; RepoRootPath defaults to koder-flow-repositories (modules/setting/repository.go:302). App metadata (usersorgsissues/ PRs) was already migrated to kdb via modules/kdb. The git content (objectsrefspackfiles) is still bare repos on disk.

Reading: the metadata plane has largely converged on kdb. The bytes plane is fragmented — Hub bytes on local FS, Flow git content on FS, kdrive bytes on MinIO — and the Koder-native object substrate that would unify them does not exist yet.

3. Proposal A — kdb-obj: object storage as a kdb-family layer

Add an object-storage layer to the kdb family (kdb-obj) that:

• exposes an S3-compatible / signed-URL API as a kdb model (same brand,

  CLI, auth, namespace, SDK as the rest of kdb);

• is internally backed by an object-native substrate — erasure

  coding, no consensus on the immutable bytes, large-extent placement, streaming IO, immutable ref-counted GC — not kdb-walkdb-buffer/ MVCC;

• coordinates with the kdb transactional core for metadataACLsigned-URL

  config (the RFC-001 pattern), so callers see one system.

kdb-obj is the Koder-native replacement for MinIO. It closes the self-hosted-first gap (§2) and lets stack-RFC-001's blob exception evolve from "MinIO forever" to "Koder object store, eventually, as a kdb-family layer." Until kdb-obj ships, MinIO/kdrive remains the backend — kdb-obj is a brandAPIsubstrate addition, not a rewrite of the transactional core.

The unifying payoff: Hub's empty Storage backend slot, kdrive's bytes, and Flow's git objects can all point at the same kdb-obj backend without any of those components changing their architecture — they already expose the metadata/bytes seam.

4. Proposal B — git storage plane (Flow off-filesystem)

A git repository is the bytes/metadata fronteira in miniature. Split it and route each half to where RFC-001 already says it belongs — the same split kdrive and Hub use:

Refinement — size-tiered objects, mirroring git's own loose-vs-pack distinction: small loose objects (commits/trees) may live in kdb core (it is good at many small objects); large blobs/packfiles go to kdb-obj.

Bonus: refs-in-kdb gives a transactional ref backend — atomic CAS ref updates across parallel pushers. That is exactly the primitive the concurrent-VCS work (§5) needs, and it falls out for free.

Feasibility in Flow (Forgejo assumes a POSIX FS at RepoRootPath), from heaviest to most pragmatic:

1. VFS/FUSE presenting kdb+kdb-obj as a POSIX tree (MS "VFS for Git"

   approach). Heavy.

2. git pluggable backends: reftable for refs (→ kdb) + partial-clone

   / promisor / custom ODB for objects (→ kdb-obj).

3. (recommended) tiered durability: Forgejo keeps operating on a local

   working copy; the durable tier is kdb (refs) + kdb-obj (packs), hydrated/persisted via a sync layer — how git alternates + remote object stores, and GitHub's DGit/Spokes replication, already work.

Phasing gate: ship refs→kdb first (cheap, unlocks concurrency), objects→tiered second (only after phase 1 proves out).

Spike validated (FLOW-147, 2026-05-24): a PoC on an isolated VM proved the refs→kdb primitive — single-statement CAS gives exactly-one-winner under 500 concurrent pushers (PASS), p99 2.7 ms vs the filesystem's 385 ms fsync tail, and the refs-in-store/objects-on-disk split works end to end. Verdict: GO on refs (productionization in FLOW-148, tiered-durability + reftable); HOLD objects until the kdb-obj decision (§7.1).

5. Proposal C — concurrent VCS layer (jj-style, git as backend)

The motivating question ("redesign git for the AI era") resolves to: the core data model (content-addressed store + immutable commit DAG + cheap branch) is timeless and should be kept. What aged badly is the shell: line-level textual diff/merge, the global mutable index, the CLI, and the absence of provenance — all of which become acute when N agents edit the same repo in parallel (the regime git was never tuned for).

Strategy: do not replace git or fork its C core to change the data model. Build a new front-end + concurrent-merge engine on top of the git object store (the Jujutsu model: git as backend, first-class committable conflicts, no index). Flow is the host; the durable tier is §4.

Implementation home — Koder KVS (products/dev/kvs). This layer is not a new component: it is the architecture for Koder KVS, the existing client-side VCS product (CLITUIdesktop/web wrapping git). KVS already ships pieces of §5 — KVS#179 conflict-as-data (= first-class committable conflicts) and KVS#180 virtual-branch-cli. The split is clean: KVS = the client/engine (this section); Flow = the host (the refs→kdb durable tier, FLOW-148). Cross-linked here + in KVS#184 so the concurrent-VCS effort does not fork into two parallel "Koder VCS" tracks.

5.1 Mechanism taxonomy

5.2 Feature → mechanism

Reading: 8/10 features are D or L (zero fork, full git interop). F is optional and singular. E appears exactly once — commutative patches — and is the only thing that would justify abandoning git as backend.

Backlog mapping (filed 2026-05-24): semantic merge → KVS#186; first-class conflicts → KVS#179 (done); per-actor staging / no-index → KVS#188; AI provenance → KVS#187; structured I/O + new CLI → KVS#189; commutative patches (the E gate) → KVS#190; SHA-256/BLAKE3 + tamper-evident provenance → FLOW-149; large blobs / chunked CAS → FLOW-148 phase 2 + kdb-obj (§7.1); refs→kdb concurrent CAS → FLOW-147 (done) / FLOW-148. Virtual branches already shipped in KVS#180.

RESOLVED 2026-05-26 — FLOW-149 (SHA-256/BLAKE3 + tamper-evident provenance): SHA-256 is the config mechanism and Flow already supports it per-repo (git ≥ 2.42); shipped the org-wide default knob [repository] DEFAULT_OBJECT_FORMAT (default sha1 — sha1↔sha256 interop still in flight, so opt-in). BLAKE3 = upstream-git matter, skipped (git has no BLAKE3 object format; no Koder fork). Hash-embedded tamper-evident provenance (F) deferred behind KVS#187's trailers/notes (D) being shown insufficient — not yet, so unbuilt.

5.3 The one deferred decision

AST-aware merge (D) vs commutative patches (E). AST-aware gives better conflict-free merges but needs a parser per language; commutative patches are language-agnostic but coarser. Start with AST-aware as a merge driver; gate the commutative-patch (E) engine behind "the driver proved insufficient." Never enter E speculatively.

RESOLVED 2026-05-25 — stay on D/L; do NOT build E (decision memo: KVS#190). D (KVS#186 Go + KVS#192 braceDartRustJSTSJavaC) and L (KVS#179 first-class conflicts, KVS#180 virtual branches, KVS#188 no-index) shipped. E's defining cost — losing native git interop (git becomes a lossy export, KVS not git would be the source of truth) — is incompatible with a git-everything Stack (Flow/Gitea, FLOW-148 refs→kdb, GitHub mirrors). Industry precedent confirms: jj (git-backed, the model §5 adopts) wins; Pijul/Darcs (patch theory) have ~zero adoption on interop grounds. The observed N-agent workload (partition-by-component via locks + AST-resolved same-file edits) is fully served by D/L. Reopen criteria (both required): (a) the Stack decides KVS — not git — is the source of truth, AND (b) a measured intra-file independent-change conflict rate stays high despite D + change-ids. The git-native middle path for the residual rebase/cherry-pick associativity pain is change-id trailers (KVS#196), not E.

6. Proposal D — physical datacenter trust tiering

Convergence's strongest operational/security argument (distinct from the developer "store once, query any way" argument): self-hosted convergence lets Koder design DC physical/security topology that cloud-renters cannot.

But §1 means it is not one box — durable state sorts into a small number of tiers with distinct physical profiles, and that is better for security because placement mirrors criticality:

Counter-instinct to encode: "more redundancy" for the consensus core means local infra redundancy (powernetworkdisk) + async DR to a second site — not spreading the Raft quorum geographically. The quorum needs low inter-node RTT; stretching it across distant sites (e.g. across EVEO DCs that do not peer directly) wrecks write latency. The blob tier is what spreads geographically (throughput/durability, not quorum).

The same metadata/bytes fronteira that shapes git, kdrive and Hub thus also shapes the physical DC: transactional core → tight bunker; bytes → spread durable tier; ephemeral → edge.

7. Open decisions / gates

1. Build kdb-obj, or keep MinIO as a permanent exception?

   Resolved 2026-06-08 (owner, k-arch→k-go) — BUILD kdb-obj (Koder-native object substrate; MinIO is the bridge, not the destination). self-hosted-first wins over focus. Pair #18 already registered (§8). Structural home: infra/data/oss becomes the kdb-obj substrate under the kdb family (NOT a standalone "kdrive" product — kdrive is the Koder Drive consumer product, products/horizontal/drive; the object store is the substrate beneath it). Now gates decision 4 (how to build) as the active sub-decision; OSS-005 identity resolves to kdb-obj.

2. AST-aware merge vs commutative patches — deferred behind the §5.3

   gate.

3. Git plane phasing — refs→kdb first; objects→tiered only after.
4. ~*kdb-obj substrate* build erasure-coded object store in Koda/Rust,

   or wrap an embeddable object store?~Resolved 2026-06-08 (owner, k-arch→k-go) — WRAP an embeddable Rust object engine, behind a swappable storage-engine boundary. Koder owns the differentiating layer (S3 API, Koder ID auth, multi-tenancyRLS, erasuretrust-tiering policy, kdb-family integration); the durable byte placement reuses a proven embeddable engine vendored in-tree (single binary, kdb-family — materially more self-hosted than the rejected external MinIO service, NOT a from-scratch greenfield). Rationale: an object store's quality IS durability → proven > greenfield (Regra 13); Rust like the rest of kdb (NOT Koda, whose GC is mid-debug — koda#810); the engine boundary keeps a future Koder-native byte substrate an optional swap, not a forced phase. Engine selection (Garage vs object_store-crate-over-durable-backend vs …) is a sub-investigation (OSS-006), not part of this decision. OSS-006 RESOLVED 2026-06-08 → ADR-001: the investigation found no single embeddable-as-a-library durable Rust object engine exists in 2026 — every finished S3 server (GarageRustFSSeaweedFS) is service-shaped (= the MinIO problem this decision rejected; Garage is also AGPL, SeaweedFS is Go, RustFS is alpha). The decision's intent (proven > greenfield, embeddable-as-library, Rust, swappable boundary) is therefore honored by composing proven Rust primitives rather than wrapping one monolith: s3s (Apache-2.0, S3 protocol front) + reed-solomon-simd (MIT/BSD, erasure primitive) + kdb itself (object→shard metadata + RLS), behind an object_store-shaped ObjectEngine trait extended with Koder's placement-policy vocabulary. reed-solomon-simd IS the proven EC primitive, so the "no greenfield byte substrate" constraint holds. RustFS watch-listed for the EC layer (rustfs-ecstore) at its 1.0. Build: OSS-007..OSS-011.

5. Trust-tiering: codify as a section here or as a runbook?

   Resolved 2026-05-24 — codified as meta/context/runbooks/datacenter-trust-tiering.kmd.

8. Amendments to normative artifacts

Owner-ratified 2026-05-24 (commit 7b59edc867):

• ✅ APPLIED — stack-RFC-001 §1 (Object/blob storage): blob exception

  evolved from "MinIO/S3-compat" to "object store; Koder-native target kdb-obj (object-native substrate, not the transactional core)", plus a substrate-exception note added to the layer taxonomy (model ≠ substrate).

• ✅ APPLIED — registries/self-hosted-pairs.md: pair #18

  MinIO ↔ kdb-obj added — status planning, substrate-distinct from row 5 (kdb-next core), gates per policies/self-hosted-first.kmd. Closes the §2 gap. koder.toml fragment pending component bootstrap.

• ✅ APPLIED — trust-tiering runbook: written at

  meta/context/runbooks/datacenter-trust-tiering.kmd (per content-location.kmd a general Stack runbook lives in meta/context/runbooks/, not infrastructure/). Encodes the §6 tiers + the "quorum tight, DR async" rule, grounded in the current single-host reality (s.khost1) with the gap→target prerequisites (a second site s.khost2 for DR is the hard blocker).

9. References

• meta/docs/stack/rfcs/stack-RFC-001-kdb-as-unified-data-plane.kmd (§1

  blob non-goal; layer taxonomy; 3 permanent exceptions)

• meta/docs/stack/registries/self-hosted-pairs.md (kdb pair; MinIO gap)
• infra/data/kdb/README.md (multi-model engine; crate list; no blob layer)
• products/dev/hub/depot/storage/storage.go,

  products/dev/hub/depot/main.go (Hub metadata→kdb, bytes→Storage iface, empty object-store backend slot)

• products/dev/flow/engine/modules/setting/repository.go (Flow git on FS;

  metadata on kdb via modules/kdb)

• policies/self-hosted-first.kmd, policies/multi-tenant-by-default.kmd
• External precedents: Jujutsu (jj — git-backend VCS, first-class

  conflicts), PijulDarcs (patch theory), CephRADOS (object-substrate unification), GitHub DGit/Spokes, MS VFS for Git / Scalar.