kdedup — `dev/kdedup`

Fast file deduplicator. Three-pass algorithm (size group → 4 KB head xxhash → full xxhash), concurrent workers, structured JSON output. Reincarnates the lost rmdup prototype from 2026-03-13, now properly homed in the monorepo.

Role in the stack

Area Sector Consumers
Foundation Linux Tools every Koder developer workstation; CI runners; bulk-data pipelines

Sibling utility to kicon (build-time icons) and kosh (Linux Shell): single-purpose Go CLI binary, shipped via Koder Hub, conformant to specs/binaries-and-cli/naming.kmd (k<slug> form).

Primary couplings

Module Nature
dev/koder-tools Same structured-output contract (applied[] / deferred[] / errors[] per RFC-001 §4)
specs/binaries-and-cli/naming.kmd Binary kdedup follows k<slug> convention
github.com/cespare/xxhash/v2 Hash function — fast non-cryptographic, ~10× faster than MD5/SHA-256
dev/kpkg (planned) One-line install: kpkg install kdedup

Public surface

kdedup [flags] [<dir>]      Scan <dir> (default: current dir)
kdedup version

Flags:
  --apply                actually remove duplicates (off by default)
  --keep <strategy>      first|newest|oldest (default: first)
  --min-size <bytes>     skip files smaller than this (default: 1)
  --workers <n>          concurrent workers (default: NumCPU)
  --format <fmt>         json|text (default: json)
  --purge                with --apply: unlink instead of XDG trash

Output: structured JSON envelope per dev/koder-tools/docs/rfcs/RFC-001-koder-tools-architecture.md §4.

Performance

Reference benchmark (2026-04-23, this commit, on ~/dev/koder/dev/eye, 2 927 files, 674 MB):

Tool Wall time Speedup vs fdupes
fdupes 2.273 s 1.0×
kdedup 0.121 s ~19×

Beats the 10× target in RFC-001 §6 by 2× on a small-to-medium tree; larger trees with more dup candidates should see proportionally better wins (the head-hash filter prunes more aggressively at scale).

Status

v0.1.0 (2026-04-23) — released.

Component Status
3-pass algorithm (size → head-hash → full-xxhash) ✅
Concurrent workers (default = NumCPU) ✅
--apply with `--keep first newest oldest` ✅
Structured JSON output (koder-tools contract) ✅
13 unit tests passing (7 hasher + 6 scanner) ✅
Single ~3 MB Go binary ✅
--purge flag (parsed; XDG trash impl pending) ⏳ v0.2
Hardlink mode (rmlint-style) ⏳ v0.3 (RFC-001 §9)
Bench script vs fdupesjdupesrmlint on synthetic 50K-file tree ⏳ v0.2
kpkg install kdedup distribution ⏳ rides on dev/koder-tools ticket KDEDUP-006 pattern

Dependencies

  • Go 1.22+
  • github.com/cespare/xxhash/v2 (only runtime dep)
  • No system deps; portable across Linux / macOS / Windows

Design references

  • RFC-001 — Fast file dedup algorithm
  • dev/koder-tools/docs/rfcs/RFC-001-koder-tools-architecture.md (output contract sibling RFC)
  • specs/binaries-and-cli/naming.kmd (binary naming convention)