Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#1953 Durable, detached workflow execution (DBOS evaluated as the engine)

Opened by nagyv · 9/2/2026

Problem

swamp workflow run --help states it plainly: "Blocks until the run completes, suspends (manual approval), fails, or is cancelled. There is no async/detached mode."

That leaves three related gaps:

  1. No durable execution. If the CLI process (or swamp serve instance) dies mid-DAG, the run is lost. The only recovery is swamp workflow resume <wf> --from <step> — manual, human-driven re-entry that depends on someone knowing which step to re-enter at and on guards correctly suppressing completed work.
  2. No detached runs. Long workflows hold a terminal open. #1507 gives cross-instance visibility of active runs, but not detachment or recovery.
  3. No step-level retry primitive. The workflow schema has allowFailure, guard, queueTimeout and concurrency, but nothing expressing "retry this step 5 times with exponential backoff". Every extension author re-implements it inside the method, or the step just fails on a transient error.

Proposed Solution

Make the workflow engine durable while keeping YAML as the authored artifact: each job becomes a durable workflow, each step a checkpointed step, with checkpoints in the repo's existing store. Concretely that would give:

  • automatic mid-DAG recovery after a crash or restart, replacing resume --from <step> as the primary recovery path;
  • a detached/async run mode, with swamp run history --active and swamp run doctor reporting against real persisted state;
  • declarative step retries (retries, intervalSeconds, backoffRate) in the step schema;
  • manual_approval expressed as a durable wait rather than suspended-run bookkeeping;
  • a natural home for concurrency/weight/queueTimeout as real queues, which would also address the per-model lock contention that currently forces fan-out methods over parallel method run calls.

Keeping YAML as the authored surface matters because the current schema is consumed statically, before any step runs: writes, platform/target/ labels for worker placement, guard CEL expressions, dataOutputOverrides (lifetime, GC, vault, vary), reports.require/skip, assert severities, and the JSON-Schema inputs block. That is what makes swamp workflow validate, swamp workflow evaluate, --last-evaluated, pre-flight checks, --fail-on, --junit, access grants and --no-supersede possible. A code-authored workflow would have to re-declare all of it as metadata, or lose it.

Alternative Considered: DBOS as the engine

I evaluated DBOS Transact (docs at docs.dbos.dev) for this. Feature fit is strong — durable steps, queues with concurrency and rate limits, durable sleep, send/recv/setEvent for human-in-the-loop, step retries with backoff, child workflows. It maps onto the existing schema almost item for item.

Three findings argue against adopting it naively:

  1. The TypeScript SDK requires Postgres. systemDatabaseUrl takes a Postgres connection string and the SDK uses a node-postgres pool. The Python SDK supports zero-config SQLite; the TypeScript one does not. For a local-first CLI on SQLite, "install Postgres to run a workflow" is a different product. Any adoption would need embedded Postgres, an optional durable mode that degrades without it, or a swamp-native SQLite checkpointer.
  2. Deno support is unverified. swamp loads extensions through an embedded Deno runtime; the DBOS TypeScript docs do not mention Deno at all. The SDK expects Node (pg, Buffer, AsyncLocalStorage, decorators, a long-lived process with an admin HTTP server). Node-compat may well carry it, but this needs a spike before anything is committed.
  3. Its upgrade model conflicts with swamp workflow edit. DBOS derives applicationVersion from a hash of workflow source and only recovers workflows whose version matches, recommending blue-green drain or DBOS.patch() markers for breaking changes. swamp's model is "edit the file, run it again" — under version-matched recovery, every edit orphans in-flight runs of that workflow. If swamp adopted DBOS, versioning would need to key off the YAML/plan hash, with an explicit story for in-flight runs across an edit.

Given (1) and (2), implementing checkpointing over swamp's existing store may be less total work than adopting the dependency, while preserving zero-infra local operation. Either way the semantics above are the ask; DBOS is one way to get them.

Also Considered: TypeScript-authored workflows

The prompt for this evaluation was that the YAML+CEL authoring surface feels dated — a 40-line LLM prompt as a YAML block scalar with ${{ data.latest(...) }} interpolation would read better as a template literal, and typed model methods would give real IDE completion instead of CEL strings that only fail at workflow evaluate time. That is a fair ergonomic complaint, but it is a separate decision from durability, and it carries the cost of the static-analysis surface described above. It is also worth weighing that agent-authored YAML is schema-validated before a step runs, whereas agent-authored TypeScript fails at runtime — inside an engine that would faithfully retry it.

If TS authoring is ever pursued, the sane shape is a front-end that compiles to the same internal plan the YAML produces, so validate/evaluate/placement/reports keep working — not a second engine. YAML-to-TS codegen is straightforward (YAML is the more restricted language); it is the static feature surface, not the transpiler, where the effort lives.

Notes

Filed from an evaluation, not from a specific failing run. Related: #1507 (active-run tracking in the control-plane store) is a prerequisite for cross-instance recovery.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/2/2026, 1:54:57 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.