Koder Transcode

Area: Media · Sector: Compute · Status: v0.1.5 (Development)

Transcoding pipeline for the Koder Stack. Responsible for converting videoaudio uploads into optimized streaming formats (DASHHLS/CMAF) with multiple renditions, codecs, and HDR formats.

Architecture

• Substrate: Rust-based job queue with in-memory (v0.1) and KDB-backed (v0.2+) storage.
• Worker (subprocess): FFmpeg-powered worker pool with Bubblewrap (bwrap) sandboxing — original v0.1 path, still supported.
• Worker (pure-Rust): pipeline::run_pipeline composes any VideoEncoder (rav1e ships pure-Rust; x264/x265 ship via cgo behind feature flags; audio wraps queued in XCODE-
  ) with Fmp4Muxer for a no-subprocess transcode flow per RFC v2 ADR-002.
• Progress: Real-time progress streaming via ProgressChannel — PROBEPACKAGEENCODE/FINALIZE stages emitted by both worker paths.

Key Features

• [x] Job queue with priority and worker capability matching (XCODE-
  ).
• [x] Sandboxed FFmpeg execution using bwrap (XCODE-
  ).
• [x] Multi-rendition codec ladder (XCODE-
  ).
• [x] DASHHLSCMAF manifest generation (XCODE-
  #007#008).
• [x] GPU acceleration (NVENCQSVVAAPI) (XCODE-
  ).
• [x] Live ingest (XCODE-
  ).
• [x] HTTP API + Prometheus metrics + OTel tracing (XCODE-
  #017#021).
• [x] fmp4 fragment muxer — pure Rust, no subprocess (XCODE-
  ).
• [x] rav1e AV1 encoder wrap, feature-gated (XCODE-
  ).
• [x] Bench harness (XCODE-
  ) — manifest + queue + muxer + encoder hot paths.
• [x] Pure-Rust pipeline composing encoder + muxer + progress (XCODE-
  ).
• [x] x264 cgo encoder wrap — pre-generated bindgen 0.71 bindings + libx264 link (XCODE-
  ).
• [x] x265 cgo encoder wrap — pre-generated bindgen 0.71 bindings + libx265 link (XCODE-
  ).
• [x] Audio encoder wraps — Opus (audiopus) + AAC-LC (fdk-aac) (XCODE-
  ).
• [x] Per-tenant rate limiter on /v1/jobs via koder_foundation_util::RateLimiter (XCODE-
  ).
• [x] Input URI scheme validation (blob:// / file:// / http(s):// only) (XCODE-
  ).
• [x] TranscodeEventSink — JobAccepted / Claimed / SegmentEmitted / Completed / Failed (XCODE-
  ).
• [x] kdb_transcode_encoder_seconds{codec, height} per-encoder histogram (XCODE-
  ).
• [x] worker.rs marked legacy; pipeline.rs is canonical worker (XCODE-
  ).
• [x] run_audio_pipeline for Spotify-side audio-only encode flow (XCODE-
  ).
• [x] Encoder bench coverage — x264x265opus/aac-lc latency (XCODE-
  ).
• [x] Tenant isolation auth — Bearer token → allowed_tenants (XCODE-
  ).
• [x] Multi-rendition fan-out via tokio task pool with semaphore (XCODE-
  ).
• [x] bin/worker.rs migrated from subprocess ffmpeg to pure-Rust ladder pipeline (XCODE-
  ).

Usage

use koder_transcode::{JobQueue, TranscodeJob, TranscodeProfile};

let queue = JobQueue::new();
let profile = TranscodeProfile::youtube_full();
let job = TranscodeJob::new("my-tenant", "blob://uploads/vid.mp4", profile);
let job_id = queue.enqueue(job)?;

Pure-Rust pipeline (RFC v2)

use koder_transcode::{
    run_pipeline, EncoderConfig, Fmp4Muxer, MuxerConfig, MuxerTrack,
    ProgressChannel, Rav1eEncoder, VideoFrame,
};

let mut enc = Rav1eEncoder::new(EncoderConfig {
    width: 1920, height: 1080,
    bitrate_kbps: 5_000, frame_rate: 30, quality: 6,
})?;
let mut mux = Fmp4Muxer::new(MuxerConfig {
    tracks: vec![MuxerTrack::av1(1, 1920, 1080)],
    fragmented_duration: 600_000,
});
let progress = ProgressChannel::new();
let frames: Vec<VideoFrame> =  vec![];
let out = run_pipeline(&mut enc, &mut mux, &frames, &progress, "job-1")?;
// out.init_segment + out.fragments → DASH/HLS CMAF segment set.

Related Modules

• data/blob: Input/output substrate.
• engines/sdk/koder_player: Consumer-side playback.
• products/horizontal/play/engine: Consumer flagship integration.