Koder Kzip — `engines/compress/kzip`

engines/compress/kzip/ · Charter stage · 2026-05-03

Purpose

General-purpose compactor for the Koder Stack. Covers single-file compression (gzip-style), multi-file archives with dedup, AEAD encryption, snapshots/versioning, FUSE mount, and cloud backends — as a single CLI + library.

Status

  • Charter accepted (RFC-001).
  • Bootstrapped from restic v0.18.1 — commit 6c182be6e, 2026-05-03.
  • Format spec at meta/docs/stack/specs/kzip/format.kmd — diverges from restic v0.18.x at §13.3-§13.6 + §14.5.
  • Build: cd engines/compress/kzip && make build → bin/kzip.
  • Run: bin/kzip version → kzip 0.2.0 (restic-fork base 0.18.1) compiled with go1.25.0 on linux/amd64.
  • Landing: https://kzip.koder.dev (live; public prd since 2026-06-10 — was dev-gated, promoted in the format-landing launch wave, ticket #034).
  • CI: .gitea/workflows/kzip-ci.yml (go-test + e2e + cross-compile pra 5 targets) desde v0.2.0.
  • Production-readiness: 5 de 6 passos da meta entregues (2026-05-19) — v0.2.0 release prep, CI gate, 1 GiB bench, soak harness, soak run de 24h iniciado (PID 3757624, log /home/koder/temp/kzip-soak-24h.log, metrics em /home/koder/temp/kzip-soak-j4V1gV/metrics.csv). Pendente: Hub publication (🔴 BLOCKED em credential — ver ticket #029) + dogfood /k-backup --engine=kzip (observation window 2026-05-19 → 2026-06-19).

Why

  • Stack ships across mobiledesktopserver/embedded; needs one compactor everywhere.
  • policies/self-hosted-first.kmd mandates progressive replacement of FOSS deps with self-hosted Koder equivalents.
  • Today's patchwork: tar+gzip, tar+xz, ad-hoc restic/borg per host. Different UX, different operator knowledge, sync breakage.

Naming

kzip — 4 chars, K-prefix family, brand score ~88. Selected over kpak (collided with kpkg), kfold, kpress, karc, kram, knot. See RFC-001 §9.

Canonical extension: .kz for single-file, .kzip for multi-file archive.

Format

Spec at meta/docs/stack/specs/kzip/format.kmd — to be drafted post-bootstrap. Format-first: stable from v1; implementation evolves underneath.

Implementation language

Pending FOSS base selection. Likely Go (if restic) or Rust (if zstd-based ground-up).

Feature list

40 features curated from comparative analysis of zstd, xz, 7-Zip, zpaq, borg, restic, dwarfs, squashfs, lz4, brotli, lzip, tar. Full list in RFC-001 §3.

Seven dimensions:

  1. Algorithms / preprocessing (LZ77 family, BWT, dicts, BCJ, delta, filter chain)
  2. Container / archive (multi-file, central directory, streaming, solid, metadata, sparse, multi-volume, append, snapshots, dedup)
  3. Performance (MT, levels, memory, SIMD)
  4. Integrity / security (CRCSHABlake3/xxhash, Reed-Solomon, AES-256 AEAD, X25519 PK, signatures, append-only)
  5. Usability (gzip drop-in, multi-language bindings, listextractverifyrandom-accessdry-run, FUSE, cloud+SSH backends, public format spec)

FOSS comparison (RFC-001 §5)

Tool Score Lang License Notes
borg 29/40 Python+C BSD Most features overall
restic 28/40 Go BSD Cloud-native; Stack-aligned (Go); recommended
7-Zip 26/40 C++ LGPL Best algorithms; no dedup
zpaq 25/40 C++ public domain Append-only versioned archive
zstd 20/40 C BSD Algorithm only; container would be ours
xz 18/40 C public domain Smaller codebase

Layout (post-#008, canonical)

Top-level Go module with module path koder.dev/engines/compress/kzip. The bootstrap-era engine/<tool>_vendor/ subdirectory was retired in ticket #008 (2026-05-13) — Koder additions sit as first-class siblings of packages inherited from restic. Full historical layout in RFC-001 §7.1; canonical layout in §7.2.

engines/compress/kzip/
├── go.mod                    ← module koder.dev/engines/compress/kzip
├── go.sum
├── Makefile                  ← `make build` → bin/kzip (GOWORK=off)
├── koder.toml
├── README.md, CHANGELOG.md, LICENSE
├── docs/
│   ├── rfcs/RFC-001-charter.md
│   └── upstream/restic-{NOTICE,CHANGELOG,README,Makefile}
├── cmd/kzip/                 ← cobra-based CLI entry
├── internal/                 ← all packages
│   ├── archiver/  backend/  repository/  crypto/  fuse/  …
│   ├── pubkey/               ← Koder addition (#005)
│   ├── filters/              ← Koder addition (#003 BCJ + delta)
│   ├── krecovery/            ← Koder addition (#007 + #009)
│   └── kzdispatch/           ← Koder addition (#010 magic-byte)
├── tests/{e2e,regression}/
└── backlog/{pending,in-progress,done}/

Backlog

Ticket Status Description
001 done Bootstrap from restic v0.18.1
002 done Landing kzip.koder.dev — live at https://kzip.koder.dev
003 done BCJ filter set (multi-arch executable preprocessing) — all phases shipped; closed alongside #015
004 done FUSE mount Linux native — exposed + tested (E2E in tests/e2e/fuse_mount_test.sh) + documented (docs/usage/fuse-mount.md)
005 done X25519 public-key crypto for repository keys — kzip key gen-identity + kzip init --recipient + global --identity; E2E in tests/e2e/backup_with_pubkey_test.sh; spec format.kmd §3.4.1; doc docs/usage/public-key-mode.md
006 done Windows snapshot mount via WinFsp — winfsp/go-winfsp pure-Go binding; cross-platform mount lands w/ no cgo on Windows (closed 2026-05-14)
007 done Reed-Solomon recovery records — v1 sidecar mode; kzip recovery {encode,verify,repair}; E2E in tests/e2e/recovery_test.sh; doc docs/usage/recovery-records.md
008 done Top-level go.mod migration — koder.dev/engines/compress/kzip; vendor subdir retired; 389 files / 1101 imports rewritten; cmd/restic → cmd/kzip; full test suite + 5 E2E + 4 regression scripts pass; regression guard tests/regression/004-top-level-module-path.test.sh enforces no return to old import path
009 done Reed-Solomon recovery records — pack-format-embedded variant (Type5 per format.kmd §13.4); `kzip backup --recovery-recordsN% + kzip recovery repair-pack id`; per-shard SHA-256 for direct repair; E2E verified
010 done Magic-byte dispatch + .kz extension unification
011 done Vendor go.mod refresh + GNU-tar dispatch fix
012 done BCJ x86 filter
013 done BCJ ARM family filters (A32 / AArch64 / Thumb-2)
014 done BCJ misc-arch filters (PPC BE/LE / SPARC / RISC-V / IA-64)
015 done kzip backup --filter=<chain> wired into pack-writer + reader — E2E byte-identical for delta:1 and bcj-x86,delta:1; Type=4 pack header per spec §13.3
016 done BCJ filter chain on tar-of-binaries — content-sniff aware path
017 done Per-chunk content sniffer + FlagFilterSkipped bit (spec §13.3.1 / §14.4)
018 done Pluggable compressor backends umbrella — internal/compressor/ registry + KZC payload envelope (spec §13.5 / RFC-002); CLI --compressor=<name>[:level] + kzip compressor list; zstd kept bare for legacy bit-compat
019 done LZMA2 backend (pure-Go via ulikunitz/xz) — child of #018
020 done Brotli backend (pure-Go via andybalholm/brotli) — child of #018
021 done Bench baseline refresh + kzip-lzma2 / kzip-brotli variants in bench/bench.sh; bench/baseline.md populated for random-100m + text-100m (2026-05-19)
022 done kzip migrate recompress — in-place per-pack re-encode to a different compressor backend, reuses prune's Repack primitive; blob IDs preserved; kzip check green before and after (RFC-002 §6 follow-up)
023 done Detached signatures (kzip sign / kzip verify) — Ed25519, kzig-v1 sidecar (.kz.sig, 101 B) per format.kmd §13.6; separate signing-identity files (.kzkey/.kzpub); --pubkey + --keyring hardening; closes RFC-001 §3 feature 33
024 done gzip-compatible single-file CLI drop-in (kzip compress / kzip decompress / implicit-dispatch + gunzip symlink) — RFC-001 §2 charter promise; internal/kzstream/, format spec §13.2 ratified-on-emission (2026-05-19)
025 done Pre-built zstd dictionary — consumer side: --dict <path> + $KZIP_DICT env var; Codec.SetDict + new DictAware interface; zstd backend implements per-call encoder/decoder with WithEncoderDict. v1 contract: file-based, same dict on read+write; in-repo persistence + trainer → #027
027 done Dict persistence in repo (restic.Config.Dict) + Go-native trainer (kzip dict train / dict inspect / dict set) — wraps klauspost zstd.BuildDict; init --dict persists, auto-loads on subsequent opens; closes #025 follow-up
026 done Framework for configurable integrity hash: internal/hashalg/ registry + SHA-256 + Blake3 (32-byte digests); Config.HashAlg persisted; kzip hash list + init --hash=<name> recorded. Full restic.Hash dispatch wiring deferred to #028 (evidence-gated)
028 done Wired internal/hashalg into restic.Hash via Design A (package-level state). kzip init --hash=blake3 now produces real Blake3 blob IDs end-to-end. Dual-repo guard via ErrHashAlgConflict. Spec divergence §14.5
029 pending Unblock Hub publication: regenerate meta/context/credentials/store-publish-key (BLOCKED on operator action — current file is binary garbage, .enc password missing)
030 done (framework) Bench rigor framework — --runs, --warmup, --cpu, --max-load flags + helpers (wait_for_quiet_machine, maybe_taskset, median_of). Per-tool N-run loop integration is sub-follow-up; today operator gets the gates + flags + scaffold for tagged-release bench prep.
031 done Fix TestStdinFromCommand* to resolve python via LookPath("python3") then fallback python, then skip cleanly. 4 tests now PASS on python3-only hosts (Debian 13, Fedora 41+, Arch).
032 done make release target — cross-compile 5 targets + SHA256SUMS manifest in one shot. bin/kzip-<target>[.exe] + bin/SHA256SUMS. make release-clean companion.
033 done Systemd-user unit + timer for the soak harness — tests/soak/kzip-soak.service (sandboxed, MemoryMax4G) + kzip-soak.timer (weekly Sunday rollover). systemd-analyze --user verify green. Operator-overridable knobs via `Environment. Install runbook in testssoakREADME.md`.

Refs

  • Charter RFC: engines/compress/kzip/docs/rfcs/RFC-001-charter.md
  • Self-hosted-first policy: meta/docs/stack/policies/self-hosted-first.kmd
  • Brand-score spec: meta/docs/stack/specs/naming/brand-score.kmd
  • Module catalog entry: row 17c in meta/docs/stack/module-catalog.md
  • Related: meta/context/commands/k-backup.md (consumer of kzip once shipped)