Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#2153 Recover interrupted workflow runs from durable checkpoints

Opened by stack72 · 9/15/2026· Shipped 9/16/2026

Problem

Swamp persists workflow-run state, supports detached swamp serve execution, and supports manual resume with swamp workflow resume --from <step>. However, when a serve instance crashes during a workflow run, the in-flight run is reaped as interrupted or failed rather than automatically recovering from its last safe checkpoint.

Recovery currently requires an operator to inspect the run, determine whether the in-flight step changed an external system, select a resume point, and rely on guards or model-method idempotency to avoid duplicate work. This makes long-running, scheduled, and unattended workflows operationally fragile.

Desired Outcome

Make interrupted workflow runs recoverable from durable step-boundary checkpoints. At run start, retain the evaluated run plan and its identity: resolved workflow graph, non-secret effective inputs, evaluated step targets, and relevant definition and extension fingerprints. Persist completed step boundaries as the run progresses.

After a crash or restart, Swamp must identify an in-flight step as unknown; it must never silently treat it as successful. Recovery can continue automatically only from a known-safe boundary. Re-executing an unknown step must require a declared idempotency guard or provider idempotency key, or explicit operator acknowledgement.

Non-Goals

This does not promise exactly-once execution. A crash can occur after an external side effect succeeds but before Swamp records step completion. The recovery model must make that ambiguity visible and preserve safe, explicit reconciliation paths.

This does not replace the existing detached serve-run behavior, manual-approval persistence, or manual resume --from flow. It extends them with safe automatic recovery for interrupted runs.

Declarative bounded step retries and backoff are a natural follow-up once attempt state and safe retry semantics are persisted.

Rationale

Workflow orchestration is a Swamp responsibility: Swamp owns the run plan, step order, lifecycle state, worker dispatch, approvals, and cancellation. Persisting and recovering this execution state lets workflows remain dependable automation without forcing every extension author to reimplement crash recovery.

Related: #1953.

02Bog Flow
OPENTRIAGEDIN PROGRESSSHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 12 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/16/2026, 6:55:00 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/16/2026, 3:46:18 PM

Sign in to post a ripple.