AI Workflow — Agent DAG / State Machine

• Area: Intelligence
• Path: services/ai/workflow
• Kind: DAG / state-machine orchestration for multi-agent / multi-step AI flows
• Status: v0.1.0 — foundation shipped (2026-05-24); YAML DSL parser, lint CLI, DOT render, Go data model. Engine production wiring (#008) closed 2026-05-28: durable PGStore, 5 per-kind executors, per-step lease + intra-run parallelism (#019), Prometheus/trace Observer backend (#020)

Role in the stack

workflow is the durable execution layer for AI flows. agents/ orchestrates a single agent loop; real production AI workloads need more — multi-step pipelines that route, retry, branch, pause for humans, and survive crashes. Today every team builds this ad-hoc with brittle state-on-disk hacks; no observability, no resume-from-failure. This sector consolidates the capability.

It is the Koder analog of LangGraph, Inngest, Temporal AI, and Restate. Distinct from services/foundation/bpm — bpm handles BPMN-style human/form-driven business process; workflow handles agent-heavy execution where the actors are LLMs, tools, and sandboxed code. Cross-sector composition allowed.

Boundary vs neighbors

• services/foundation/bpm is the business-process sibling (humans + forms + approvals); workflow is the AI-execution sibling.
• services/ai/agents is the single-agent loop — workflow steps invoke agents as actors.
• services/ai/sandbox, services/ai/tools, services/ai/runtime, services/ai/gateway are step backends (code, tool, model calls).
• services/ai/trace receives per-run + per-step spans.
• infra/data/kdb-doc (state) and infra/data/kdb-blob (large outputs) are storage layers.

Features (v1 target)

• YAML wire-format workflow definitions + Go typed step SDK
• 9 step kinds: llm, tool, code, agent, subflow, human, branch, parallel, aggregate
• CEL expressions for conditional edges
• Durable runs (kdb-doc state, kdb-blob payloads, lease-based worker assignment)
• Crash recovery with idempotency keys
• Per-step retry policies (exponential backoff, max attempts/duration)
• Human-in-the-loop pauses with signal-based resume + Koder ID notification routing
• Fan-out (parallel/branch) + fan-in (aggregate) + sub-workflow composition
• kworkflow CLI (lint, render DOT, trigger)
• Run-monitoring UI at workflow.koder.dev (graph + step inspect + signal form)
• Per-tenant concurrent-step quota
• Trace span per run + child span per step attempt

Primary couplings

RFC and bootstrap

• RFC: workflow-RFC-001-foundations.kmd — accepted 2026-05-09
• Bootstrap ticket: services/ai/backlog/done/136-workflow-bootstrap.md
• Implementation tickets: 001 (DSL), 002 engine core — DONE 2026-05-25 (backend/internal/engine: durable RunStepResult, in-memory Store + per-run leaseTTL crash-recovery, StartTickCancel, retry policy w/ exponential backoff, edge-guard advancement, 7 tests under -race; Store+Executor interfaces let kdb-dockdb-blob + real per-kind backends swap in via #008), 003 human-in-the-loop — DONE 2026-05-25 (human step pauses the run + releases the lease; Signal validates choicesrequired-fields + requireall consensus + resumes with the payload as step output; CheckTimeout auto-decides via `defaultchoice or fails — 5 tests; Koder ID notification routing/role-expansion/authz is a tail), **004 fan-out/fan-in/subflow — DONE 2026-05-25** (aggregate.go: fan-in join aggregateReady with all/any/quorum:N modes — an aggregate runs only once its upstream join condition is met; all-with-fail falls out of the engine failing on any failed step; CollectInputs merges predecessor outputs for the aggregate Executor; fan-out parallel/branch is the existing multi-successor enqueue; subflow dispatches to the Executor like any backend kind — 6 new tests, engine suite now 18 green under -race; real nested-run recursion/cancellation-propagation/state-scoping/quota/composition-docs deferred to #008), **007 CEL expression validator — DONE 2026-05-25** (internallintexpr.go: **dependency-free** CEL-subset validator wired into Lint over every edge.if + branch config.expression → WF-INVALID-EXPRESSION with 1-based column; tokenizer + balanced-bracket/terminated-string check + identifier classification (declared roots state/steps/inputs, known-global arity, open-ended method/field access) catches -for-= typos, undeclared vars, wrong arity; **owner decision 2026-05-25 chose dependency-free over cel-go** to keep the module at 1-dep — full type-check + runtime eval stays an engine concern for #008 behind the same code; lint suite 15 green under -race), **006 Go SDK typed authoring — DONE 2026-05-25** (backendpkgauthoring: fluent Builder — Define().OwnerTenant().Input().Step(name, spec).Edge()EdgeIf().Build() — with typed constructors for all 9 step kinds (LLM`ToolCodeAgentSubflowHumanBranchParallelAggregateAggregateQuorum) whose signatures enforce required config at compile time (no nil model); Build() runs lint.Lint (incl. the #007 CEL validator) then marshals the canonical dsl.Workflow, round-tripping through dsl.Parse+lint.Lint clean; 7 testsexamples under -race, 5 worked godoc examples. *laced in-module per the pkgkdb→enginessdkgokdb (SDKGO-004) pattern* promotion to `enginessdkgoworkflow is #009), **008 CEL guard evaluator slice — DONE 2026-05-25** (internalengineeval.go: dependency-free recursive-descent parser + evaluator for the #007 CEL subset — field/index access, comparison/logical(short-circuit)/arithmetic, in, ternary, size/has, literals; EvalGuard is now the engine's **default** GuardFunc so non-empty if: edges actually route, fail-closed on error; the engine builds the {state, steps, inputs} scope per run via evalScope — steps.X.output←History, inputs.←new Run.Inputs; 4 tests incl. real routing on a step output + a seeded input; engine suite 22 green under -race`. *008 persistent Store — DONE 2026-05-26 (internal/engine/pgstore.go PGStore: Postgres-backed engine.Store over the pkg/kdb SDK — owner-ratified over kdb-doc which isn't running yet; a run persists as a single JSONB doc + lease columns, Lease is an atomic conditional UPDATE for crash-resume, Migrate via the kdb Migrator; 3 integration tests against real PG — round-trip, lease CAS w/ expiry, full engine run persisted — on the s.khost1.dev-linux-workflow VM; kdb-blob output-refs + per-step lease + metrics/trace remain; subflow Executor — DONE 2026-05-26 internal/engine/subflow.go SubflowExecutor: a subflow step resolves a named child workflow from a WorkflowSource and runs it to completion on a nested engine (sync, child inherits parent tenant, depth-bounded cycle guard, async rejected), returning the child's terminal {state,status}; delegates leaf kinds to an inner executor; 8 tests, completes the #004 tail — the llmtoolcode/agent executors need their backends running; observability seam — DONE 2026-05-26 internal/engine/observer.go: an Observer interface (StepStartedStepFinished+durationStepRetryRunFinished) the engine calls from TickrunStepcompleteStepCancel, NopObserver default, RunFinished fires once per terminal status; 5 tests — the real Prometheus`servicesai/trace` backend wires behind it), 010 state-mutation model — DONE 2026-05-25 (owner-ratified explicit output_to* dsl.Step.OutputTo names the state var a step's output writes on success — written before outgoing-guard eval so the step's own edges can route on it; no output_to ⇒ no write, wire-format additive; lint WF-INVALID-OUTPUT-TO validates the identifier + rejects reserved roots; StepSpec.OutputTo SDK modifier; RFC-001 amended; reducers set: deferred), 005/006 + the integration remainder of 008 + 009 pending.
• 008 engine production wiring — CLOSED 2026-05-28: all five per-kind executors (llmtoolcodeagentsubflow) production-wired, retryErrTerminal classification, CEL guard, aggregateparallel joins, persistent PGStore, Observer seam — plus the two final engine-level carve-outs landed this session. 019 per-step lease + intra-run parallelism — DONE 2026-05-28 (Store gains LeaseStepRefreshStepReleaseStepExpiredStepLeases; MemStore + PGStore workflow_step_lease migration v2, inclusive-expiry atomic upsert; dsl.Workflow.MaxConcurrent + engine.Options.MaxConcurrentSteps; Engine.Tick now dispatches a batch of ready steps on per-step-leased goroutines with heartbeat, applying completions sequentially under the run lease — serial-by-default preserves backward-compat; barrierpeak-concurrency + serial-default + MemStore-semantics + parallel-failure tests under -race, TestPGStoreStepLeaseCAS/Takeover green on real PG at s.khost1.dev-linux-workflow; residual = formal wall-clock ≥3× perf-baseline number). 020 engine Observer Prometheus + trace — DONE 2026-05-28 (MetricsObserver dependency-free Prometheus text exporter — koder_workflow_{runs,steps,step_retries}_total + {run,step}_duration_ms histograms + workers_busy gauge — mirroring the AI-sector no-client_golang pattern; TraceObserver over a dependency-free TracerSpan seam for servicesaitrace; MultiObserver composes SSE hub + metrics + trace; wfsrv --metrics-addr :9330 serves `metrics, port reserved in koder-service-ports; --trace-otlp-endpoint` flag; residual = live OTLP collector verification).
• 018 SSE Last-Event-ID resume + admin-role gating — DONE 2026-05-28 (split from #014). wire.Event.ID per-run monotonic id; EventLog interface (MemEventLog default + PGEventLog workflow_events table); the Hub is the single sequencing point (assigns id under lock — correct under #019 parallel dispatch — seeds from MaxID for restart-safe monotonic ids, persists then fans out, drops post-close duplicates); writeSSE emits id:, handleStream honors Last-Event-ID (header + query fallback) replaying only missed events (no snapshot) and deduping live overlap. Admin-role gating: RoleChecker interface + RoleTenantOwnerRoleWorkflowAdmin, requireRole after the tenant check (cross-tenant stays 404 per R2) → cancel needs owner|admin, replay needs admin, denial 403 WF_FORBIDDEN; nil checker = gating off (boot-safe). Tests -race (resumerestart/role matrix) + TestPGEventLogRoundTrip PASS on dev-linux-workflow. Lesson: kdb Migrator _migrations is a global per-DB version counter → PGEventLog.Migrate uses direct idempotent DDL. Residual: live Koder-ID RoleChecker client + OTLP, retention prune.

Self-hosted-first analysis (5 gates)