Pipe — Workflow Orchestration
- Area: Data Platform
- Path:
platform/pipe - Kind: Workflow / pipeline orchestration engine
Role in the stack
Pipe is the workflow and pipeline orchestration engine. It is where long-running, multi-step, asynchronous processes are defined and executed: ETL jobs, data migrations, CI pipelines, batch processing, webhook fan-outs. Consumers describe a pipeline as a DAG of steps; Pipe schedules, retries, and observes them.
Primary couplings
| Consumer | Relationship |
|---|---|
platform/kdb |
State storage (pipeline runs, step outputs) |
data/mq |
Job queueing between steps |
platform/ci |
CI pipelines use Pipe as the execution engine |
| `observe |