AI Dataset — Versioning + Splits + Schema + Dedup

• Area: Intelligence
• Path: services/ai/dataset
• Kind: Dataset foundation (versioning + splits + schema validation + dedup; analog of HF Datasets / W&B Tables / DVC)
• Status: v0.0.1 — sector bootstrapping (2026-05-09)

Role in the stack

dataset consolidates ML data plumbing. ML reproducibility requires versioned datasets — without it, "trained on dataset X" is meaningless. products/horizontal/libras is currently designing its ingest pipeline in isolation; if a second project starts the same way, they reinvent splits, schema validation, dedup and lineage from scratch.

It is the Koder analog of HuggingFace Datasets, Weights & Biases Tables, DVC, lakeFS and Pachyderm — built thin on top of kdb-blob (content-addressable) + kdb-meta (manifests + lineage), with a kdataset CLI for the human path.

Boundary vs neighbors

• services/ai/training is the primary downstream consumer (mandatory input).
• services/ai/embed is used internally for semantic dedup.
• services/ai/extract is a curated source upstream.
• Generic ETL is out of scope (separate problem space).

Features (v1 target)

• Content-addressable manifests (versions share blobs)
• Schema validation (strict / permissive / repair modes)
• Deterministic splits (hashed partitioning; stratified + group-aware)
• Dedup (exact-hash + semantic-cosine via embed)
• Lineage tracking (every transform records inputs + params)
• Cross-tenant explicit grant
• Parquet + JSONL + Arrow IO
• kdataset CLI as first-class human path

Primary couplings

RFC and bootstrap

• RFC: dataset-RFC-001-foundations.kmd — accepted 2026-05-09
• Bootstrap ticket: services/ai/backlog/done/123-dataset-bootstrap.md
• Implementation tickets: services/ai/dataset/backlog/pending/{001..005}

Self-hosted-first analysis (5 gates)