Internal Service mTLS & East-West PKI — the trust fabric for service-to-service auth

Document format reference: meta/docs/stack/policies/document-format.kmd

Status: RATIFIED 2026-06-01 (owner). §5 recommendation adopted as decided: two-layer model — WireGuard transport (Layer 1, already canonical) + Koder ID-rooted x509 SVIDs with the CA key sealed by the KMS (Layer 2, Option C) — with Option A (WG-only) as the interim so consumers ship now and harden later. §7 resolved as recommended: (1) Option C (Koder ID issuer, not a new CA component); (2) WG-now, SVID-additive (defense-in-depth, not a prerequisite); (3) short TTL (hours), consumer-pull rotation like SA tokens; (4) reuse Koder ID's keys service; (5) short-TTL-only revocation first (no CRL/OCSP). Implementation tracked: services/foundation/id/engine#204 (SVID issuance) + services/crypto/kms#004 (CA-key custody). Consumers: FLEET-039 (mTLS slice), kdb-RFC-008 §8.1.

Origin: the multi-region campaign (owner decision 2026-06-01, /k-go fleet) surfaced that two consumers — Fleet hub↔member mTLS (infra-RFC-004 §6.4 / FLEET-039) and kdb cross-region mTLS (kdb-RFC-008 §8.1) — both need an internal service-identity / cert story the Stack did not have. Decided once, here.

1. The gap

The Stack has a clear story for north-south (browser-facing) TLS and for transport privacy, but no story for east-west per-service identity:

So when Fleet's hub must authenticate a member cluster as a workload identity (not just "reachable"), or when a kdb primary must mTLS to a secondary-region standby, there is no Koder-native way to issue/rotate the certs. ACME is for public domains; the signer is for legal documents; the KMS does envelope, not issuance.

2. Constraints (from existing policy)

• R1 — self-hosted-first. No external CA / cert-manager / step-ca as a dependency

  (self-hosted-first.kmd). The issuer must be a Koder component.

• R2 — reuse-first. Do not stand up a new trust root if an existing authority can

  host the capability (reuse-first.kmd). Koder ID is already the Stack's identity authority (it mints SA tokens + holds JWKS).

• R3 — one transport fabric. Cross-host/cross-region private traffic already rides

  the WireGuard overlay (observe-RFC-001). Do not grow a parallel private transport.

• R4 — Quality > Speed (long-term). Prefer per-service, short-lived, rotatable

  identity over coarse/static trust, when the extra effort is bounded (stack-principles.kmd §2).

3. The two layers (they are complementary, not competing)

The core insight: transport privacy and service identity are different problems, and the Stack already solves the first.

• Layer 1 — Transport (private fabric): WireGuard. Already canonical

  (observe-RFC-001). Gives mutual peer auth (Noise static keys) + encryption between hostsDCs. Trust granularity = *er hostpeer* not per service.

• Layer 2 — Identity (per-service mTLS): x509 SVIDs. Needed only when authz must be

  per service/workload (e.g. "this is the fleet hub", "this is the *kdb primary in region us*"), beyond "some process on a trusted host". This is the layer the Stack lacks.

4. Options for Layer 2 (the decision)

Option A — WireGuard-only (no app-layer mTLS)

Ride WG for everything; the peer identity is the trust boundary. App authz stays at the SA-token (JWT) layer over the WG link.

• + Zero new code; reuses proven infra; matches observe-RFC-001 exactly.
• − Trust is per-host, not per-service: any process on a peer host inherits the

  peer's trust. No cert-level workload identity, no per-service revocation. Fine for coarse host↔host; weak for a multi-tenant control plane where which service matters.

Option B — New dedicated CA component (services/crypto/ca)

A purpose-built internal CA issuing short-lived SVIDs; CA key sealed by KMS.

• + Textbook; clean separation.
• − Violates R2 (reuse-first): a second identity root alongside Koder ID +

  a new component to buildoperatesecure. Two places that answer "who is this".

Option C — Koder ID as the SVID issuer (CA key KMS-sealed) ⭐

Extend Koder ID's keys service with a cert-issuance endpoint: a service presents its existing Koder ID service-account credential (the trust it already has, FLEET-020) and receives a short-lived x509 SVID (SPIFFE-style identity). The issuing CA private key is sealed by the KMS (envelope / systemd-creds), so custody stays in the service KMS is built for. Consumers refresh SVIDs the same way they refresh SA tokens.

• + Single identity root (Koder ID already mints SA tokens — now also x509);

  reuse-first (R2); KMS does key custody (its job); per-service identity + short TTL + rotation (R4); the SA↔SVID mapping is 1:1 with the auth model that exists.

• − Extends Koder ID's surface (a CSR-signing endpoint + CA-key custody wiring);

  Koder ID becomes more security-critical (it already is — it's the auth root).

Option D — External (step-ca / Vault PKI / cert-manager)

• Rejected by R1 (self-hosted-first).

5. Recommendation

Adopt the two-layer model: WireGuard transport (Layer 1, already done) + Option C (Koder ID-rooted SVIDs, KMS-sealed CA key) for Layer 2.

Rationale (R1–R4): it keeps one transport fabric (WG), one identity root (Koder ID — no second authority, R2), puts key custody in the KMS (its purpose), and gives per-service, short-lived, rotatable identity (R4) — the structurally-best form, not a v1-to-redo. Option A is the interim (it ships now with zero new code); Option C is the target (per-service identity). They compose: A is literally "Layer 1 only", C adds Layer 2 on top of the same WG fabric.

Identity naming

SPIFFE-style URI SAN: spiffe://koder/<area>/<sector>/<instance> (e.g. spiffe://koder/infra/net/fleet/hub-br-sp, spiffe://koder/infra/data/kdb/primary-us), 1:1 with the Koder ID service-account it is issued against. Reconcile exact format with specs/naming/ at implementation.

6. Immediate path for the blocked consumers (no waiting on Layer 2)

Both consumers unblock now on Layer 1 and upgrade to Layer 2 when it ships:

• Fleet hub↔member (FLEET-039 mTLS slice). Now: members reachable over the WG

  overlay; trust = WG peer + the existing SA bearer (FLEET-035/020). Later: swap the bearer for a Koder ID SVID (mTLS) once the issuance endpoint exists. The FLEET-039 ticket's "mTLS slice" becomes "wire the SVID client once stack-RFC-009 Layer 2 lands".

• kdb cross-region mTLS (kdb-RFC-008 §8.1). Now: WAL streaming rides the WG

  overlay between regions (same as telemetry). Later: per-node SVID mTLS for the replication channel.

This means neither campaign is blocked on this RFC — they ride WG today; Layer 2 is an additive hardening, not a prerequisite.

7. Open questions (for ratification)

1. Ratify Option C (Koder ID as issuer) vs Option B (dedicated CA)? — recommend C.
2. Is WG sufficient as Layer 1 for cross-region prod, or do we want app-mTLS even

   inside the WG tunnel (defense-in-depth)? — recommend WG-now, SVID-additive.

3. SVID TTL + rotation cadence (recommend hours, consumer-pull like SA tokens).
4. Does the SVID issuer reuse Koder ID's keys service or a new keys-sibling?
5. Revocation model — short TTL (no CRL) vs CRL/OCSP (recommend short-TTL-only first).

8. Non-goals

• Replacing Jet/ACME for public TLS (north-south is solved).
• Replacing the signer (legal document signatures — different domain).
• A service mesh / sidecar proxy (this is library-level mTLS, not a mesh).

9. Next steps (post-ratification)

1. Owner ratifies §5 + §7.
2. Open id#NNN — Koder ID SVID issuance endpoint (CSR-sign, CA key KMS-sealed).
3. Open kms#NNN — seal/rotate the SVID CA key (if not covered by existing envelope).
4. Update FLEET-039 (mTLS slice) + kdb-RFC-008 §8.1 to consume the SVID client.
5. Register the SVID identity scheme under specs/naming/.