Relationships
#2518 serve: a suspended run with an expired gate from a previous serve instance cannot be cancelled by any path
Opened by randybias · 9/25/2026· Shipped 10/5/2026
Problem
A suspended run whose approval gate has timed out, owned by a serve instance that has since restarted, cannot be cleared by any path. It stays suspended / waiting_approval forever, and the dashboard shows it as a live run with no action available.
Measured (20260923.231117.0)
Run 910a5683 of a gated workflow was started through serve on 2026-09-22 22:37Z. The gate has timeout: 3600, and serve has restarted several times since.
- Approve / reject: refused. The gate is past its timeout (
evaluateApprovalTimeout), and the run is omitted fromworkflow.approvals: the dashboard Approvals view is empty. - Cancel through serve:
swamp workflow cancel <wf> --run 910a5683… --server wss://…→404 {"status":"not_found","message":"No active workflow-run with id 910a5683… in this serve instance"} - Cancel locally, after
datastore sync --pull:Run 910a5683… belongs to a serve instance and cannot be cancelled locally. Use --server to cancel it - Result: the design doc's remedy for an expired gate ("the run stays suspended — cancel it to clear it",
design/primitives/workflows.md) is unreachable. The run shows in the dashboard as a live run inwaiting_approval, with nothing to click.
Ask
- Let
cancel --serveract on any suspended run in the shared run store, not only runs in this instance's active registry. Suspended runs have no live executor to lose. - Show an expired gate in the dashboard as expired, with a Cancel action, instead of omitting it from Approvals while the run list still says
waiting_approval.
Shipped
Click a lifecycle step above to view its details.
randybias commented 10/4/2026, 10:14:45 AM
Still present on 20260929.002922.0-sha.55e2ef29 (serve, token mode, s3-datastore), 9 days later. New measurements, 2026-10-04 ~10:00Z:
- The dashboard Approvals view lists 4 runs of one workflow under "Awaiting resume": ad562537-dbbf-4fbf-ad25-f56dc3c81b5b, 6675231a-3de1-474f-82df-026a3d72a19e, 9226b00f-186e-4add-b83b-ea9f9ffb67a3, 05542f40-7ffb-431c-8cf8-090af75ed5f0. Each shows "Approved — awaiting resume" and a Resume button. Started 2026-09-25; serve has restarted many times since.
swamp workflow history get ad562537-... --json(through serve):status: "running", startedAt 2026-09-25T09:22:28Z. So the record says running, while the search row says suspended + awaitingResume.swamp workflow cancel <wf> --run ad562537-...(admin):404 No active workflow-run with id ... in this serve instance.swamp run doctor --json(through serve):active 0, stale 0, orphanedWorkflowRuns 0. Doctor does not see them.swamp run gchelp: "Running and suspended runs are never deleted regardless of age."
So no supported path removes them: cancel needs the owning instance, doctor does not classify them, and gc skips them. Resume is not acceptable: it would execute a 9-day-old mutating run. They stay in Approvals forever, next to real gates.
Ask: either let doctor (or cancel --force for an admin) mark a run whose owning instance is gone as cancelled/orphaned, or have serve reconcile at boot every non-terminal run it does not own.
hammz commented 10/5/2026, 10:37:38 PM
Thanks @randybias for reporting this! We shipped: The reported state (suspended, expired gate, restarted serve) is already fixed on main by the swamp-club#2514 persisted-run cancel. This plan fixes what still strands runs: the run index losing updates under concurrent saves, orphan finders that trust that index and a seven-day cutoff, and server cancel refusing a running record whose owner is gone. The issue's dashboard ask (expired gates shown as expired, with a Cancel action) moved to swamp-club#3045.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.