Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
Assigneeshammz

Relationships

#2443 workflow recover: a changed definition leaves the interrupted run with no way to recover or resume

Opened by hammz · 9/23/2026

Problem

When swamp workflow recover refuses an interrupted run because the workflow definition changed after the run started (fingerprint mismatch), it tells the operator to run swamp workflow resume --from <step> instead. That command does not accept interrupted runs, and no other command does either, so the run can be neither recovered nor resumed. The only way forward is a fresh run.

Every route, tried against an interrupted run with the installed binary:

Command Result
swamp workflow resume <wf> --from <step> --run <id> (the suggested remedy) --from requires a failed run, but run <id> has status "interrupted"
swamp workflow resume <wf> --from <step> No failed runs found for workflow "<wf>". The latest run is interrupted (<id>). Recover the interrupted run with 'swamp workflow recover <wf>'. (sends the operator back to recover)
swamp workflow resume <wf> --run <id> Run <id> is not suspended (status: interrupted)
swamp workflow recover <wf> --acknowledge-unknown Still refused: the mismatch check runs before the acknowledgement check

Where

  • The suggested remedy: src/domain/workflows/recovery_assessment.ts:65 (assessment reason), src/cli/commands/workflow_recover.ts:135 (--assess-only output) and :143 (refusal).
  • --from accepts only failed runs: src/domain/workflows/suspended_run_resolver.ts:119-123 and src/domain/workflows/execution_service.ts:2558-2562.
  • The mismatch check comes before the acknowledgement check: src/cli/commands/workflow_recover.ts:142-152.

Relation to swamp-club#2431

Today every workflow that uses an inputs.* expression hits this, because the drift check compares the stored fingerprint of the evaluated workflow against the raw definition and so always reports a change. swamp-club#2431 fixes that false positive. After it ships, the dead end remains for runs whose definition really did change.

Decision needed

Pick the path forward for an interrupted run whose definition changed:

  • let resume --from <step> accept interrupted runs and re-enter at the named step against the current definition, or
  • change the refusal so it names a path that works (for example, start a new run), and fix the assessment text to match.

Reproduction

  1. Get an interrupted run of any workflow (for example, crash swamp serve mid-run and restart it; the boot reaper marks the run interrupted).
  2. Edit the workflow definition.
  3. Run swamp workflow recover <wf> (needs the swamp-club#2431 fix; before that the command crashes).
  4. Follow the suggested swamp workflow resume --from <step> command; it is rejected as shown above.
02Bog Flow
✓OPEN✓TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNEDCLASSIFICATION

Closed

9/29/2026, 4:58:30 PM

No activity in this phase yet.

03Sludge Pulse
hammz assigned hammz9/29/2026, 4:51:17 PM
Editable. Press Enter to edit.

hammz commented 9/23/2026, 10:18:42 PM

Update from swamp-club#2431: the UX review there blocked shipping the resume --from advice, since recover now works and operators would see it for the first time. Once #2431 lands, both drift refusals (definition changed, and runs recorded before definition fingerprints) end with: start a new run with 'swamp workflow run '. The --assess-only output no longer suggests resume --from or --acknowledge-unknown when the definition check fails, and the design doc and manual say the same.

What remains for this issue is the design question: should an interrupted run whose definition changed be continuable, for example by letting resume --from accept interrupted runs? If so, update the refusal text in recovery_assessment.ts to point at it.

hammz commented 9/29/2026, 4:58:30 PM

Closing as fixed by swamp-club#2431 (PR #2599, commit 20b7797a, released in v20260923.230324.0).

That change took the second option above. When the workflow definition changed since an interrupted run started, both refusals from 'swamp workflow recover' now name a path that works: start a new run with 'swamp workflow run '. They no longer point at 'resume --from', which rejects interrupted runs. When the definition check fails, '--assess-only' also stops suggesting '--acknowledge-unknown' and 'resume --from'.

We decided not to take the first option, letting 'resume --from' accept interrupted runs against a changed definition. For an interrupted run whose definition really changed, starting a new run is the intended path.

Sign in to post a ripple.