Relationships
#2548 workflow cancel on a live local run overwrites the owning process's final run record with a stale snapshot
Opened by hammz · 9/25/2026· Shipped 9/28/2026
Summary
The offline path of swamp workflow cancel (src/cli/commands/workflow_cancel.ts) loads the run record, then cancelRun() calls killProcessTree(run.pid), which sends SIGTERM and waits for the owning process to exit, then calls run.cancel(reason) on the record it loaded before the kill and saves it.
A local swamp workflow run process handles SIGTERM as an abort: it runs cleanup mode (swamp-club#1785), marks in-flight steps failed (cancelled), runs always/completed cleanup steps, and saves its final record. The cancel command then saves its stale snapshot over that record, so the persisted run loses every step outcome recorded after the snapshot was read, including cleanup steps that actually ran.
Steps to reproduce
- A workflow with a forEach step build over [1, 2, 3], concurrency 1, each iteration running sleep 5, and a step rollback with dependsOn build, condition type failed.
- Start swamp workflow run in the background, wait about 2.5s, then run swamp workflow cancel with a reason.
- The owning process's live output shows the cleanup-mode evaluation (rollback skipped (dependency), job failed, workflow cancelled).
- swamp workflow history get shows the persisted record instead: run cancelled with the cancel_reason tag, but job main pending, and steps build (the unexpanded forEach template name, not build-1..3) and rollback both pending, with no error or skipReason.
Observed with 20260925.180024.0-sha.e08f1d5b. The record is the snapshot saved at run start, before forEach expansion; the owning process had not saved again before the cancel command read it.
Expected
An offline cancel of a run whose owning process is alive and exits after SIGTERM keeps the owning process's final record: either re-read the run after the kill and cancel only if it is still non-terminal (adding the cancel_reason tag), or skip the write when the reloaded run is already terminal. A run whose owner was SIGKILLed or never wrote a terminal state still gets the cancelled status.
Context
Found while reproducing swamp-club#2543 (never-started dependencies stay pending after a cancel or --timeout). Once #2543 marks never-started steps terminal, this overwrite would still discard those marks when the cancel comes from another process.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.