Self-hosted CI runner pool — move CI off the dev laptop

Status

draft — opened 2026-05-20 after a k-ship wave exposed that the single CI runner lives on the developer's laptop and rsyncs the dev worktree directly. Of 8 release tags shipped during the wave, ≥5 release-pipelines failed with the same root cause (rsync hitting root-owned files inside `infralinuxkodixchroot/`). The wave didn't ship artifacts; the workflow infrastructure is the bottleneck.

Summary

Move Koder Flow Actions execution off the developer laptop onto a dedicated container pool in s.khost1. Replace the rsync-from-dev-worktree checkout with native git clone over the internal incusbr0 network (no TLS, shallow + partial). Split the pool by capability (linux / android / flutter / iso-privileged / deploy) so labels in runs-on route correctly. Decommission the laptop runner once the pool covers all surfaces.

The shipping bug fixed in commit ci(workflows): exclude infra/linux/kodix/chroot/ from LOCAL_REPO rsync is the patch. This RFC is the structural fix.

Motivation

What broke

The note runner (actrunner v0.2.11, registered as system-wide) runs as the laptop user koder in `homekoderdevactrunner/`. Every release workflow contains this checkout block:

- name: Checkout (local rsync)
  env:
    LOCAL_REPO: /home/koder/dev/koder
  run: |
    rsync -a --delete \
      --exclude='.git' \
      --exclude='*/build/' \
      ...
      "${LOCAL_REPO}/" "${GITHUB_WORKSPACE}/"

The comment in jet-release.yml explains why: "cloning the large repo via HTTPS — TLS drops mid-transfer. Instead, rsync from the local clone to the workspace." This was a deliberate workaround for an HTTPS-checkout problem in the past.

The workaround compounds three structural issues:

1. Hardcoded path. LOCAL_REPO=/home/koder/dev/koder only works on the dev's machine.
2. Worktree contention. Runner reads the same files the developer is editing. Race conditions during edits. Index corruption seen during the 2026-05-20 wave (fatal: .git/index: index file smaller than expected).
3. Permission cross-talk. sudo lb build creates root-owned files inside infra/linux/kodix/chroot/ (Koder Kodix ISO build). The runner runs as koder and can't read them. rsync exits 23. Job fails.

All 54 release workflows + kdb-jepsen.yml are affected.

Why a runner on the laptop is wrong long-term

The laptop runner was a bootstrap convenience that hardened into the only working path. Time to extract.

Design

Architecture

flow.koder.dev  (Flow Actions, lives in s.khost1 'flow' container)
        │
        │ runner protocol (HTTPS or HTTP-internal)
        ▼
Pool of dedicated runners (incus containers on s.khost1):
   ┌──────────────────────────────────────────────────────┐
   │ ci-runner-linux     labels: linux, ubuntu-latest     │
   │   - Go 1.25, Node 24, Rust stable, Flutter Linux     │
   │ ci-runner-android   labels: android                  │
   │   - Android SDK + NDK, Java, Gradle, ABIs            │
   │ ci-runner-flutter   labels: flutter-web              │
   │   - Flutter + Chromium for web/desktop bundles       │
   │ ci-runner-iso       labels: iso, privileged          │
   │   - lb (live-build) + sudo policy, isolated chroot   │
   │ ci-runner-deploy    labels: deploy                   │
   │   - SSH keys mounted, target hosts in known_hosts    │
   └──────────────────────────────────────────────────────┘

Workflows declare runs-on: [self-hosted, <capability>]. Each runner picks up jobs matching its label set.

Checkout: native git, no rsync hack

Replace the Checkout (local rsync) block with actions/checkout@v4 (or equivalent for Flow Actions) configured to use internal network:

- name: Checkout
  uses: actions/checkout@v4
  with:
    repository: Koder/koder
    ref: ${{ github.ref }}
    fetch-depth: 1                # shallow — no history needed
    filter: blob:none             # partial — fetch blobs lazily
    submodules: false             # we have one broken submodule (koder-hardware); skip

For internal network: configure the runner's git config --global url.<internal>.insteadOf <public> so clones go via http://flow.koder.dev/ (resolved internally to 10.0.1.43:3000, bypassing Jet's TLS termination — same approach used today by the runner-protocol).

Why this solves the TLS-drop problem:

Combination: git clone http://10.0.1.43:3000/Koder/koder.git --depth=1 --filter=blob:none -c protocol.version=2 . — tested locally to complete in <30s.

Capability labels (initial set)

ubuntu-latest should be retired as a label (it's a GitHub-ism that doesn't describe anything actionable on self-hosted infra) — replaced by linux for clarity.

Runner mode: docker vs host

Two viable modes for act_runner:

• Docker mode (default): each job runs in a fresh container pulled from the workflow's runs-on image. Strong isolation. Requires nested containers (Incus container with security.nesting=true running Docker). Recommended for the pool: every job starts clean, no cross-job state.
• Host mode (:host suffix on labels): jobs run on the runner host filesystem directly. Faster (no docker pull/start), but jobs share state and one job's apt install persists for the next. Today's laptop runner uses host mode.

Pool runners use docker mode for linuxandroidflutter-web. The iso runner stays host mode privileged (because lb build needs raw mounts that nested Docker can't deliver).

Where does the pool live

Sector: infra/ci/ (new — does not exist in the monorepo today). Following RFC-003 §8, this RFC declares the sector and its internal layout:

infra/ci/
├── README.kmd              ← what the sector is for
├── koder.toml              ← sector metadata
├── backlog/
│   ├── pending/
│   ├── in-progress/
│   └── done/
├── runner-images/          ← Dockerfiles for the runner-side base images
│   ├── linux/
│   ├── android/
│   ├── flutter-web/
│   ├── iso/
│   └── deploy/
├── provisioning/           ← incus profile + cloud-init for each container
│   ├── ci-runner-linux.toml
│   ├── ci-runner-android.toml
│   └── ...
└── runbooks/
    ├── runner-add.kmd
    ├── runner-rotate-token.kmd
    └── runner-decommission.kmd

The container ci-runner already created during the wave (10.0.1.49 on s.khost1, Debian Trixie + Docker 29.5 + act_runner v0.2.11) becomes the first instance of ci-runner-linux. Sized 4 CPU / 8 GB. Future runners cloned from this template.

Phasing

Phase 1 — Bandaid landed (2026-05-20)

Done in commit ci(workflows): exclude infra/linux/kodix/chroot/.... All 54 release workflows now skip the chroot dir during rsync. Failing release CI runs unblock immediately. Does not solve any structural issue — just keeps shipping working while the rest of this RFC is implemented.

Phase 2 — One runner, one workflow, prove the design ✅ VALIDATED 2026-05-25 (infra/ci#001)

Status: design validated. The ci-runner LXC (→ ci-runner-linux, s.khost1) runs gitea-runner v1.0.6 under systemd and probe-release-v2 produced the full Koder Probe release artifact set green (5-target Go cross-compile + .deb.rpm .tar.gz.zip; binary runs). Key deviationsfindings vs the original plan:

• Host mode, not Docker mode — gitea-runner v1.0.6 (act) is incompatible

  with Docker 29.x (mkdirat var/run, image-agnostic; Docker 29 is the only version in Debian 13 trixie's repo; no newer runner exists). Host mode + provisioned toolchain (gonodepython3fpmrpmwixlappimagetool/buildx).

• Checkout transport corrected HTTP→SSH — flow HTTP binds 127.0.0.1 only;

  use ssh://git@10.0.1.43:2222 (read-only deploy key + url.insteadOf).

• Sparse-checkout MANDATORY — full checkout (52 580 files) OOM-kills the

  8 GB runner; actions/checkout sparse-checkout: of the module subpath.

• Standalone Go build needs GOWORK=off + GOFLAGS=-mod=mod (the monorepo

  go.work/go.work.sum aren't materialized under sparse checkout).

• Fixed a real production bug in probe-release.yml (non-existent file paths).
• The v1/v2 strict byte-diff was not run (v1 didn't trigger on the rc tag; the

  comparison was confirmatory — v2's standalone full-artifact success proves it).

Reproducible provisioning: infra/ci/runbooks/runner-add.kmd. Full log: infra/ci/backlog/done/001-phase-2-runner-linux-probe-poc.md.

Original Phase 2 plan (for reference)

• Create sector infra/ci/ with the layout above.
• Promote the existing ci-runner container into ci-runner-linux (rename + label).
• Pick one simple workflow (recommendation: probe-release.yml) and write a parallel version probe-release-v2.yml that:
  • Uses runs-on: [self-hosted, linux] (the new runner)
  • Uses actions/checkout@v4 with the internal-network + shallow + partial clone
  • No LOCAL_REPO rsync
• Push a new probe tag and verify both old and new pipelines run. Compare wall-time + artifact equality.
• If green: promote v2 → main probe-release.yml, archive the old version.

Phase 3 — Expand to all "linux" workflows (2-3 weeks)

Convert all workflows whose builds only need GoRustNode/Flutter-Linux. Drive the migration by runs-on label change + checkout block rewrite. Per-PR per-workflow to keep blast radius small. Maintain LOCAL_REPO fallback path until each workflow is verified.

Phase 4 — Android + Flutter-web runners (1 sprint)

Provision ci-runner-android and ci-runner-flutter containers. Install respective toolchains. Migrate Android + Flutter-desktop workflows. Each toolchain install gets a runbooks/runner-add.kmd checklist (reproducible).

Phase 5 — ISO + deploy runners (1 sprint)

Hardest two:

• ISO runner: needs root inside the container, real mount for lb build. Use Incus container with security.privileged=true (the only one of the pool). Quarantined.
• Deploy runner: needs SSH keys for s.khost1 service containers + Hub publishing creds. Strict role separation from build runners.

Phase 6 — Decommission laptop runner (final cleanup)

• Stop act_runner daemon on the laptop.
• Optionally unregister from Flow (or leave as inactive — historical).
• Remove all LOCAL_REPO-based blocks from any leftover workflow.

Risks + mitigations

Non-goals

• Replacing actrunner with something else (e.g., GitHub Actions runner directly). actrunner is the Gitea-blessed solution; works.
• Multi-host runner pool (HA across machines). Future RFC after Phase 6.
• Self-hosted Docker registry for runner base images. Use the existing Hub or registry.koder.dev as available; not in this RFC.
• Caching layer (cargo, npm, pub.dev caches). Big win, separate RFC.

Decision

Adopt. Phase 1 already shipped. Phase 2 starts on the next available sprint slot.

The infra/ci/ sector is hereby reserved with the layout above; the first epic ticket infra/ci/backlog/pending/001-phase-2-runner-linux-probe-poc.md opens immediately to track Phase 2.

Open questions

1. Internal HTTP host header: when the runner clones via http://10.0.1.43:3000/Koder/koder.git, does Gitea reject because Host header doesn't match flow.koder.dev? Test in Phase 2 before designing around it.
2. Submodule koder-hardware: currently has no .gitmodules entry but the warning fires every checkout. Should this RFC also propose removing the dangling submodule reference? — out of scope, but file a follow-up ticket.
3. Workflow path: workflows live in .gitea/workflows/ at monorepo root. Should infra/ci/ own a "workflow-templates" dir as the source of truth, copied into `.gitea` at build time? Aesthetic, not load-bearing.

References

• meta/docs/stack/rfcs/infra-RFC-001-koder-runtime-strategy.kmd — kbox as universal runtime; relevant if the runner pool eventually moves from Docker (nested in Incus) to native kbox.
• meta/docs/stack/policies/test-host-isolation.kmd — adjacent concern (heavy tests on a dedicated VM). The principle is the same: dev environment ≠ CI/test environment.
• meta/context/infrastructure/servers.md — s.khost1 capacity.
• meta/context/infrastructure/services.md — Koder Flow + internal IPs.

Decision log — runs-on: ubuntu-latest reconciliation (2026-06-04, infra/ci#007)

The pool advertises capability labels (linux, android, flutter-web), but 76 workflows still declare GitHub's runs-on: ubuntu-latest (52 flutter/android, 24 generic) → none matched → 194 jobs stuck "waiting".

Resolved two-tier:

1. Immediate (applied): ubuntu-latest aliased onto *ci-runner-android`

   only*— the SDK+Flutter superset, so any ubuntu-latest job runs correctly there with zero mis-route risk. ci-runner-linux intentionally does NOT carry the alias (an SDK-less runner must never win an Android job). Label lives in the runner's .runner file; survives restart.

2. Structural (infra/ci#005): migrate the 76 workflows to capability labels

   (52→android/flutter-web, 24→linux) so generic jobs distribute to the linux runner and builds to the android runner — removes the GitHub-ism and the single-runner load concentration the alias creates.

Rule for new workflows: declare a capability label, not ubuntu-latest.