Protocol First

mandatory

Sub-policy de `reuse-first` para wire formats e protocol clients cross-cutting (koder_ipc, koder_chat, koder_store, observability envelopes, kpkg, federation protocol). Herda decision tree, protocolo pré-Write e promotion pipeline da meta; acrescenta apenas as regras categoricamente distintas: RFC ratificado antes do mainline, version negotiation no envelope, BC por N versões, schema evolution (add OK; rename/remove = major), cross-implementation tests.

Sub-policy of reuse-first. Inherits the decision tree, the pre-Write protocol and the promotion pipeline. Do not redeclare them here. This file contributes only the rules that are categorically distinct for wire formats and the protocol clients that speak them.

Scope

Applies to any code in:

  • `enginessdkkoder_ipc

proto/* — observability envelopes (logs / traces / metrics).

  • products/dev/hub{proto,kpkg}/** — Hub federation protocol and .kpkg package format.
  • servicesengine/proto/** — service-tier RPC contracts. Koder services default to protogRPC over RESTJSON; the canonical pattern is services/foundation/id/engine/proto/koder/id/{oauth,identity,session,auth,admin,keys,invites,saml}/v1/ — one <sub-service>.proto defining the requestresponse messages plus an optional <sub-service>_storage.proto for persistence-layer types, with `option go_package = "koder.devservicepkgprotosub-servicev1;sub-servicev1"`.
  • **/wire*.proto — any package that defines a cross-process schema.

A protocol client is in scope on two axes simultaneously: this sub-policy (wire format) and runtime-lib-first.kmd (lib mechanics). Where the two contracts apply, both must be satisfied.

Out of scope: in-process function calls, intra-module data structures, internal serialisation that never crosses a process boundary.

Categorical rules

1. Ratified RFC before mainline

A wire format or protocol does not ship from a single PR. It lands as RFC first.

  • New wire format → new RFC under meta/docs/stack/rfcs/<area>-RFC-NNN-<slug>.md with status Accepted before any code under lib/wire/ is merged.
  • Significant evolution of an existing wire format (new RPC, new message, new envelope field with semantic impact) → either an RFC amendment or a new RFC depending on scope. Flag-level changes to existing messages do not need an RFC; field additions to existing messages might (see §3 schema evolution).
  • The RFC carries the design rationale, the wire-level encoding, the version-negotiation strategy, the BC commitment, and the test matrix.

2. Version negotiation in the envelope

Every protocol envelope MUST carry an explicit version. Patterns by transport:

  • gRPC / protobuf: a version field in the request envelope, AND service/method names that include the major version (HubV1.GetPackage vs HubV2.GetPackage). This is the protobuf-canonical way and survives reverse-proxy renaming.
  • HTTP/JSON: Accept-Version: <semver-range> request header; server responds with X-Version: <chosen>.
  • Custom binary: first 2 bytes of every frame encode (major, minor).
  • kpkg: the kpkg.toml declares format-version = "1.0".

Server and client announce their supported version range and pick the highest mutually supported. A version mismatch produces a structured error with the supported ranges, not a generic 400/500.

3. Schema evolution rules

For every message / record / table:

Change Required version bump
Add an optional field none (forward-compatible)
Add a required field major (consumers without it break)
Rename a field major
Remove a field major (after deprecation cycle in §4)
Change a field's type major
Change a field's semantics (without renaming) major; anti-pattern, prefer rename
Change default value minor (document in CHANGELOG; observable behaviour change)
Add a new RPC / endpoint minor
Remove an RPC / endpoint major (after deprecation cycle in §4)

The rule of thumb: adding optionality is non-breaking; subtracting or restricting is breaking.

4. Backwards compatibility for N versions

The wire-format contract: client at version N MUST interoperate with server at version N − k for k = 0..2 (default N=2 for production protocols). A protocol with stricter or looser BC declares the value of N in its RFC.

Operationally:

  • A breaking change at version N lands behind a flag for at least one minor cycle before becoming default.
  • The server answers an old client with a structured protocol-version-too-old error including the minimum supported version, never with a generic failure.
  • Deprecated symbols / fields stay in the schema for the BC window; they may be marked deprecated = true but remain serialised and accepted.

5. Cross-implementation tests

Where multiple implementations of the same protocol exist (typical for Koder protocols: Go server + JS browser client + Python script + Rust embedded client), a cross-implementation test suite is mandatory:

  • Round-trip: each pair (impl-A → impl-B) exchanges a representative message set; both sides decode without error.
  • Compatibility matrix: client at v(N-2) ↔ server at vN; client at vN ↔ server at v(N-2). Both work for the BC window.
  • Fuzzing: feed each implementation a corpus of malformed and adversarial inputs; expected output is a structured error, never a panic.

The test suite lives in a dedicated repo path (e.g. engines/sdk/koder_ipc/tests/cross-impl/) and runs in CI on changes to any implementation.

6. Wire-format anti-patterns

The following are forbidden; reviewers reject PRs that introduce them:

  • Untyped envelopes (e.g. Map<String, Any> as the top-level message). Always typed.
  • String-encoded enums without a registered enum type. Use protobuf enums, TS literal unions, Rust enum.
  • Server-side boolean flags that change wire shape depending on user role. Roles authorise; they don't reshape the wire.
  • In-band schema (sending the schema with each message). Schema is out-of-band, versioned, and discoverable.
  • Optional-typed fields with implicit defaults that vary by language (e.g. JS undefined vs Go zero-value vs Rust Option::None interpreted differently). Defaults are explicit per the schema.

7. Self-describing artefacts

Every wire-format release ships with a machine-readable schema artefact:

  • protobuf .proto files in a versioned directory.
  • JSON Schema for HTTP/JSON protocols.
  • Rust / Go / TS / Python bindings auto-generated from the schema (see build-tooling-first.kmd §4 self-stamping for the binding-generation contract).

A protocol client's runtime version metadata includes the schema version it was generated against.

8. Promotion criterion (overrides the meta default)

For wire formats, the meta's "≥ 3 consumers" rule changes to:

  • A protocol becomes shared (lifted to engines/sdk/) on its first cross-process consumer. Reason: a wire format defined inside one product becomes a de facto contract the moment a second process speaks it; the cost of moving it later is much higher than UI widget migration.
  • The accompanying RFC (per §1 above) is the formalisation step.

Audit

./protocol-first-audit.sh (initial in Phase 2; substantive in Phase 4):

  1. Structural (Phase 2): validate inherits_from; verify referenced specssibling exist; verify each `enginessdkprotocol has a proto or wire` directory and an RFC link in its README.
  2. Behavioural (Phase 4):
    • Walk all .proto files; flag missing version fields in top-level request/response messages.
    • Walk all lib/wire/ directories; flag any Map<String, Any> / Object / interface{} envelope.
    • Parse RFCs; assert every RFC labelled protocol has a § "BC commitment" with an explicit value of N.
    • Run the cross-implementation test suite if present; CI gate fails the build if absent.

Cross-references