AI Synth — Audio Synthesis Foundation

• Area: Intelligence
• Path: services/ai/synth
• Kind: Audio synthesis foundation (TTS + cloning + music + SFX)
• Status: v0.1.0 — foundation landed 2026-05-24. HTTP daemon koder-synth + CLI ksynth + voice registry (5 built-ins seeded) + 4 routes (ttsmusicsfxclone). Default provider is the deterministic stub (silent-WAV generator) so the API contract is exercisable without GPU dependencies; Piper adapter is a typed stub awaiting synth#004. Consent flow validates token shape. synth#019 (2026-05-28, /k-go drain) shipped the consent-validator seam (`internalvoiceconsent.go): ConsentValidator interface + FormatValidator (v0 default) + HTTPValidator (validates against the id consent service per a published wire contract — accepts only valid && scope=oice-clone && !expired`, fails closed on transportnon-200); Registry.Clone validates through it and Registry.Delete revokes (both now ctx-aware); wired in cmd/koder-synth via [consent] service_url. Issuer side (id POST /v1/consent/validate + /revoke + token minting) tracked as id/engine#188; durable persistence (#017 kdb-doc voices, #018 kdb-blob audio) remains, behind the same build-tagged kdb pattern as tools#006.

Role in the stack

synth is the symmetric pair of services/ai/voice (STT). Without it, Talk Mode in products/horizontal/talk is a half-loop — the user is heard but the answer comes back as text. Narration, audio branding, accessibility (screen reader for visually impaired), in-product tutorials, agent loops with audio responses are all blocked.

It is the Koder analog of ElevenLabs (TTS + cloning), SunoUdio (music), and Stability Audio (SFX) — self-hosted via Coqui XTTS / Piper / AudioCraft on GPU runtime, with proxy fallback to ElevenLabsSuno through services/ai/gateway when local quality is insufficient or capability gaps remain.

Boundary vs neighbors

• services/ai/voice is the STT (input) sibling. Future RFC may unify under audio with audio.stt / audio.tts namespaces.
• services/ai/video may reuse synth for audio-track generation in v2.
• Audio editing/mastering and live streaming are explicitly out of scope.

Features (v1 target)

• TTS: Piper (CPU baseline, fast) + Coqui XTTS (GPU, multilingual + cloning capable)
• Voice cloning: Coqui XTTS with explicit consent capture flow
• Music: AudioCraft MusicGen up to 30s
• SFX: AudioCraft AudioGen up to 10s
• Inaudible watermark on every output (deepfake mitigation)

Primary couplings

RFC and bootstrap

• RFC: synth-RFC-001-foundations.kmd — accepted 2026-05-09
• Bootstrap ticket: services/ai/backlog/done/119-synth-bootstrap.md
• Implementation tickets: services/ai/synth/backlog/pending/{001..005}

Self-hosted-first analysis (5 gates)

Recent ships

• Observability foundation (#016 part 1) — /metrics Prometheus scrape endpoint + /readyz readiness probe + 5 declared metrics: koder_synth_api_requests_total{endpoint, status}, koder_synth_api_latency_seconds{endpoint} (10 buckets, 10ms → 30s), koder_synth_api_audio_seconds_synthesized_total{backend} (billing input), koder_synth_api_errors_total{code} (SYNTH-* code label), koder_synth_api_ready (01 mirror). metricsMiddleware labels by chi route pattern (RouteContext().RoutePattern()) to avoid voice-ID cardinality explosion. Readiness v0 = Provider != nil && Voices.Count() > 0; full worker-capability heartbeat lands with SYNTH-002+. OTLP traces + structured logs + cachequeue/watermark metrics + Grafana dashboard + alerts.yaml + drain-time metric moved to SYNTH-020.
• Content-addressed cache part 1 (#013) — internal/cache/ ships the Key + Hash() canonical envelope (SHA-256 over sorted-JSON: endpoint, textprompt, voiceid, lang, format, speed, pitch, durations, samplerate, ssmlnormalized, watermark_tenant), Envelope struct, Store interface (GetPutTierLen), and LRU implementation (FIFO eviction, cap≤0 disables). Handler instrumented across TTSMusicSFX: cache lookup before provider call, hit returns cached audio + skips audio-seconds counter (avoids double-billing); miss synthesizes + Put. Multi-tenant isolation via WatermarkTenant in the key — tenants A and B with identical inputs hash differently because each gets its own watermark embedded. Music seeded calls bypass cache (per-call seed makes the request unique). Response envelope gained Cache *CacheInfo{hit, key, tier} field (omitted when no cache wired). 2 new metrics: koder_synth_cache_hits_total{endpoint, tier} + koder_synth_cache_misses_total{endpoint}. cmd/koder-synth defaults to in-process LRU at cap 1024; [cache] config section knob in_process_cap operator-tunable. 6 cache unit tests + 2 handler integration tests. Redis hot tier + kdb-blob cold tier + DELETE invalidation by voice_id + cache:"skip" bypass moved to SYNTH-021 (the cold tier blocked on SYNTH-018).
• Voice design API part 1 (#007) — POST /v1/synth/design ships an attribute-driven voice creation path orthogonal to cloning. Registry gained SourceDesigned enum value + Attrs{Gender, Age, Pitch, Style, Accent, Dialect, Seed} struct (persisted on Voice.Attrs); voice IDs prefix vd_. Critical contract: Design() does NOT require a consent token (no real-person reference, no legal exposure), and that's pinned by regression test. Designed voices share the cloned-voice tenant-scoped store so visibility rules (owner-only by default) are inherited automatically. Handler ships DesignRequest{Name, Lang, Attrs, Seed, Provider} + DesignResponse{VoiceID, PreviewURL, AttrsResolved} (PreviewURL slot reserved for SYNTH-022's OmniVoice preview synthesis). 5 unit tests + 2 e2e tests cover roundtrip, no-consent contract, missing-field rejection, list visibility, tenant isolation, end-to-end design→tts roundtrip. OmniVoice backend routing for vd_* IDs + preview URL synthesis + deterministic seeded regen + auto-design fallback on empty TTS moved to SYNTH-022 (blocked on SYNTH-006 OmniVoice backend).
• Fixed-duration TTS scaffold (#010) — internal/duration/ ships an Estimator function type + DefaultEstimator (chars-per-sec baseline: English 13, Portuguese 14, Spanish 16, French 17, default 14) + ResolveSpeed(text, lang, durationS, speed, est) that honors the [0.5, 2.0] feasibility window. Out-of-range duration → SYNTH-VALIDATION-DURATION-INFEASIBLE-001 with the feasible interval (e.g. 5.00..20.00s) in the message so clients surface a usable hint. TTSRequest.DurationS knob (json duration_s); when both durations + speed are provided, duration wins and AudioEnvelope.Warnings carries `"speedignoredduetofixedduration". Warnings field threads through a new runTTSWithWarnings → runTTSImpl` indirection so cache hitmiss paths both surface it. Real time-stretch (soxrubberband) + OmniVoice native handoff + test-render estimator + per-language config table moved to SYNTH-023 (blocked on SYNTH-004/006 real backends). 7 unit tests + 4 e2e cover baseline rates, passthrough, override, infeasibility, empty text, lower bound, range validation.
• Non-verbal markup parser part 1 (#008) — internal/ssml/nonverbal.go ships the frozen 7-marker tagset (laughter, sigh, confirmation, question, surprise, dissatisfaction, breath) + bracket-form ([name] strict ASCII identifier regex guards against false-matching Markdown links) + SSML element-form (<koder:nonverbal type="name"/> self-closing + paired) parsers, both returning a unified []Token stream. Unknown markers reject with SYNTH-VALIDATION-UNKNOWN-NONVERBAL-001 whose message lists the accepted set so clients surface a usable hint. provider.TTSRequest.NonverbalTokens field + Capabilities.NativeNonverbal flag — backends advertise whether they consume the stream directly. Handler parses both text and ssml fields, propagates the unified stream via internal converter toProviderTokens (boundary that keeps ssml ↔ provider import cycle out). 9 unit + 4 e2e tests cover frozen-set contract pin, single/multi markers, Markdown false-positive guard, SSML self-closing + paired, unknown rejection with accepted-set message. SFX clip stitching for non-native backends + OmniVoice native passthrough + variant randomization + docs surface moved to SYNTH-024 (blocked on SYNTH-005 SFX pack + SYNTH-006 OmniVoice).
• SSML phoneme override parser part 1 (#009) — sibling to #008. internal/ssml/phoneme.go ships the W3C <phoneme alphabet="…" ph="…">word</phoneme> element parser with frozen 3-alphabet set (ipa, cmu, pinyin) + PhonemeToken{Alphabet, PH, Word} envelope + ParsePhonemes() returning (tokens, phonemes, error). Regex uses (?s) flag so Unicode word payloads (Chinese chars in Pinyin overrides) parse correctly. Token stream gained TokenPhoneme kind (Token.Value carries word fallback, envelope slice carries alphabet+ph). Unknown alphabet → SYNTH-VALIDATION-UNKNOWN-ALPHABET-001 whose message enumerates the accepted set. provider.TTSRequest.PhonemeTokens field + Capabilities.NativePhonemeAlphabets []string — backends declare which alphabets they consume ph for directly (OmniVoice all 3, Piper IPA via espeak-ng, Coqui IPA via tokenizer; conversion table from CMUPinyin → IPA lands with the renderers). Handler propagates via toProviderPhonemes mirror of the SYNTH-008 converter. 8 unit + 3 e2e tests cover frozen-set pin, all 3 alphabets including Unicode word, unknown rejection with accepted set message, multiple elements in one body, empty-PH fallback, plain text passthrough. Backend renderers + IPA conversion table + bracket-shorthand auto-detect in plain text + fixture AB audio QA moved to SYNTH-025.
• TTS engine benchmark suite part 1 (#015) — bench/tts/ ships the corpus + runner + report scaffold. Corpus is 45 fixed utterances across 5 languages (pt-BR, en-US, zh, ja, hi) × 3 buckets (shortmediumlong) × 3 utterances each, committed as corpus/<lang>.jsonl. bench/tts/run/main.go replays the corpus against a running daemon's /v1/synth/tts, captures wallms (client-side end-to-end), serverms (latency_ms from envelope), audios (`durations), and RTF per sample; supports -langs filter + -warmup knob + -backend label. Output: per-backend results.json. benchttsreportmain.go` consumes one or more results.json and emits Markdown with: runs metadata + per-(backend, lang, bucket) summary (p50p95 wall, p50 RTF, mean audio s) + overall ranking by median RTF + errors section. 5 unit tests pin section presence + column count + ranking ordering + error surfacing + percentile/mean edge cases. End-to-end smoke against the stub provider: 9 samples, report tables populated correctly. MOS proxy (Whisper BLEU) + GPU memory probe (nvidia-smi pmon) + cost per minute + CI gate (block on >20% RTF regression) + self-hosted-pairs registry auto-update + ElevenLabs external row moved to SYNTH-026 (each blocked on real backends + Whisper service + GPU LXC).
• TTS streaming output part 1 (#012) — POST /v1/synth/tts:stream ships chunked audio frames via sentence-based pipelining over the existing Provider.TTS. internal/streaming/chunker.go Splits on a multi-language terminator set (. ! ? ！ ？ 。 ।) with trailing-fragment preservation and double-terminator collapse; : and ; deliberately excluded. Two output modes negotiated via Accept: default chunked binary (audio/wav today, opus once SYNTH-027 wires it) and text/event-stream SSE base64 frames data: {seq, audio_b64, is_final, format, duration_s, provider}. Backpressure via request context — loop checks tctx.Err() per chunk so a closed client doesn't keep the backend hot. All SYNTH-008009010 gates (non-verbal, phoneme, duration) carry over and reject upfront BEFORE streaming headers ship (no "200 then SSE-error" anti-pattern). 9 unit tests for the chunker (LatinCJKHindidouble-terminatorwhitespace) + 5 e2e tests for the handler (SSE monotonic seq + is_final on last only; chunked binary content-type + ≥88-byte body for 2 chunks; validation rejection before headers; SYNTH-008 carry-over; client-cancel timeout pin). Coqui XTTS native token streaming + Opus codec + FBL <300ms gate + watermark spread-spectrum + worker queue NATS frame multiplexing + cancels counter moved to SYNTH-027 (blocked on SYNTH-004 real backends + SYNTH-002 worker queue).
• Capabilities discovery + per-key rate limit (#028, #029 — /k-evolve batch) — GET /v1/synth/info (anonymous; root router; sibling of healthz + /readyz) returns InfoEnvelope{Provider, Capabilities, Version{Binary, RFC}, Cache{Enabled, Tier}, NonverbalSet, PhonemeAlphabets} so multi-backend clients discover what each daemon supports without reading source. Frozen SYNTH-008 nonverbal + SYNTH-009 phoneme alphabet slices surfaced verbatim — 3 e2e tests pin envelope shape + frozen-set equality + cache wiring reflection. server.BuildWith(deps, mw, BuildOpts{RateLimitPerMin, RateLimitBurst}) adds an optional per-key token-bucket throttle (golang.orgxtimerate) keyed on auth.Identity{TenantID, UserID}. Middleware mounted AFTER auth (so Identity is resolved) and BEFORE metrics (so throttled requests don't pad requests_total). Throttled requests respond 429 with SYNTH-QUOTA-EXCEEDED-001 + Retry-After: 1 and tick a new koder_synth_api_throttled_total{tenant} counter; no audioseconds nor billing counters move. `[auth] perkeyratepermin + perkey_burst` in koder-synth.toml; 00 disables (v0 default). 3 e2e tests (burst budget honored, perMin=0 transparent, typed code + Retry-After on 429). Limiter map unbounded — janitor goroutine deferred ifwhen needed.
• Real consent validation + per-tenant clone retention (#019, /k-go id 2026-05-31) — completes the voice-clone consent loop. The HTTPValidator (consent.go) validates clone tokens against the Koder ID consent service (POST /v1/consent/validate + /revoke; accepts only valid && scope==voice-clone && !expired; transport-failnon-200 fail closed); Registry.Clone validates, Registry.Delete revokes. The route is config-driven (no dev-only flag): real validation activates by setting [consent] service_url in the deploy. New: per-tenant retention — a cloned voice expires clone_days after its last use (sliding window), per-tenant override via [retention] per_tenant_days; enforced on read + renewed on Get + lazily reaped (no background sweeper; forward-compatible with the future kdb-doc backing). clone_days=0 (default) keeps the prior never-expire behavior; designedbuilt-in voices are exempt. Voice.{LastUsedAt,ExpiresAt}, Registry.SetRetention. 7 retention unit tests. Issuer side is id#188 (live). Remaining for full e2e: the Talk consent client UX that mints a token via the id issuer (POST /v1/consent) — products/Talk surface, not synth-owned.