Observe OTLP Fabric — the deployed observability subsystem

• Area: Observability (infra/observe/)
• Path: cross-component (apm + mon + log + dash + OTel Collector), LXC observe (10.0.1.27)
• Kind: Self-hosted OTLP observability fabric — the deployed realization of policies/observability-first.kmd
• Status: LIVE 6/6 (2026-05-29); all 3 signals E2E-verified — traces→apm, logs→log, metrics→kdb-ts (kdb-gateway)

Role in the stack

This is how Koder telemetry actually flows in production — the concrete fabric behind the observability-first policy (dimension D8 of the Architectural Fitness rubric). Every Koder component instruments via a thin SDK binding that emits OTLP; an OTel Collector front-door fans each signal out to the self-hosted per-signal backend. One wire (OTLP), N backends.

  Go services        ──┐                         ┌── traces  → apm  (:7880, /api/v1/otlp/traces, SQLite)
  (engines/sdk/go/obs) │                          │
                       ├─ OTLP ─► OTel Collector ─┤── logs    → log-platform (:8106, /v1/logs, PG koder_log@kdb)
  Flutter apps         │  (:4317 gRPC / :4318     │
  (koder_kit           ┘   HTTP, front-door)      └── metrics → kdb-gateway (:7890, /v1/metrics, kdb-ts sled)
   KoderTelemetry)                                    dashboards: dash (:7810)

All on LXC observe (10.0.1.27, co-located v1). Ports in registries/koder-service-ports.toml; per-service keys in credentials/.

What is LIVE (2026-05-29)

The instrumentation contract

specs/observability/instrumentation-contract.kmd — one contract, N bindings:

• C1 structured log schema · C2 metric naming + cardinality deny-list ·

  C3 implicit trace/tenant context · C4 W3C propagation · C5 allow-list PII redaction · C6 OTLP export.

• Go binding engines/sdk/go/obs — thin layer over the OTel Go SDK;

  full C1–C6; E2E verified (cmd/obs-smoke → collector → apm).

• Dart binding engines/sdk/koder_kit KoderTelemetry — per-signal OTLP

  export (config-gated otlpEndpoint); C1–C6 + allow-list redaction; 22 unit tests. Both bindings emit identical OTLP — no split-brain.

Backends "speak OTLP" natively (apm /api/v1/otlp/traces, log /v1/logs) rather than the collector carrying custom translators — reuse-first, and the backends also accept gzip (Content-Encoding) for external clients. The collector→backend loopback hops use compression: none deliberately.

Primary couplings

• engines/sdk/go/obs + engines/sdk/koder_kit (KoderTelemetry) — the producers.
• OTel Collector — front-door / fan-out (config-as-code in

  infra/observe/observability/deploy/otelcol-config.yaml).

• kdb — two roles: (a) log-platform's store = PostgreSQL @10.0.1.20:5433

  (DB koder_log); (b) the metrics TSDB = kdb-gateway (kdb-next crate, sled single-node) co-located on the observe LXC, speaking OTLP /v1/metrics + Prometheus remote_write/read. The gateway IS the metrics backend — no custom ingest service (reuse-first).

• policies/observability-first.kmd — the normative source this realizes.

SDK adoption — the conformant path (OBS-062)

The fabric is only useful once real services emit through it via the shared binding, not hand-rolled telemetry. The Go binding (engines/sdk/go/obs) provides the single conformant entry point:

• obs.NewHTTP(logger).Middleware — a net/http wrapper that, per request,

  emits all three signals correlated by trace_id from one place: a span (continuing the upstream W3C trace), RED metrics (koder_<svc>_http_requests_total{method,route,status_class} + a latency histogram ..._http_request_duration_seconds{method,route} — C2.1) and a structured access log, plus a per-request panic boundary (R1.5: panic→500 + structured error). route is the mux pattern (bounded), never the concrete path (C2.2).

• obs.Histogram — the missing RED Duration instrument, same cardinality

  guard as Counter/Gauge.

Pilot: koder-notify (2026-05-30). First service to adopt it end-to-end — ad-hoc slog + a local Prometheus /metrics package were removed; the mux is wrapped by the obs middleware and the domain counters (notifications_sent/failed_total{channel_type}, channels_active) moved to obs.Counter/Gauge. The 3 signals were verified live: trace in apm (service=koder-notify), log in log-platform (fields.trace_id), RED metrics in kdb-ts.

2nd adopter: koder-symbol-store (2026-05-30, OBS-072). The build-id symbol store service (below) was instrumented with the same binding from day one, confirming the pattern generalizes — its 3 signals land in the stack with no bespoke telemetry.

Log transport — the missing receiver. The binding emits logs as JSON to stdout; the deployed collector had no stdout/file receiver, so SDK logs never reached log-platform. Fixed by adding a journald receiver (scoped per unit) to the collector's logs pipeline: stdout → journald → json_parser + flatten(fields) + severity_parser → OTLP → log-platform. Each service adopting the binding adds its unit to that receiver's units: list (now koder-notify.service + koder-symbol-store.service). Traces and metrics need no such step (the binding pushes them over OTLP/gRPC directly).

Build-id symbol store (OBS-072). A sibling debuggability service rather than part of the OTLP fabric proper: koder-symbol-store (:7930, observe LXC) keeps stripped-artifact debug-info keyed by GNU build-id behind the storage-agnostic symbols.Store interface (FSStore now; blob-backed later, same wire). CI uploads via koder-symbols extract --store-url … --key …. Resolution is a service capability (slice 4a): POST /v1/symbolicate {build_id, pcs} returns fn (file:line) server-side, with an LRU cache of resolvers so a burst of crash queries against one artifact does not re-fetch/re-parse its DWARF. apm (slice 4b, DONE 2026-05-30) is a thin client that POSTs raw frames + build-id — it never needs the symbols package or the debug-info. apm stores crashes (crash_events + POST /api/v1/crashes) and symbolicates on read (GET /api/v1/crashes/{id} via its internal/symbolize client → /v1/symbolicate), falling back to raw frames if symbols are unavailable. Verified live: a stripped-binary crash (0x401106) shows up in apm as add /tmp/fixture.c:1. The apm web UI has a Crashes tab (list + symbolicated detail, deep-link #crash-<id>); fixing it surfaced a pre-existing gap — the UI never authenticated, so every `apiv1