Relationships
#4883 serve SIGTERM skips the failed-gated cleanup job of a run the user cancelled before the shutdown
Opened by zack · 11/18/2026· Shipped 11/18/2026
Description
When a user cancels a run and serve then gets SIGTERM while the run's cleanup is still going, serve skips the run's failed-gated cleanup job. The run still ends cancelled as if its cleanup had finished, and serve logs no cut-off.
The shutdown drain is meant to settle a run the user cancelled first, never interrupt it, and wait for its cleanup within the cleanup grace (src/serve/shutdown_drain.ts:49, 163-166). The same cancel with no SIGTERM runs the cleanup job.
Steps to reproduce
- Use a workflow where job A has a long
workstep and an in-jobalwayscleanup step that takes 8 s. Job BdependsOn: Awithcondition: failedand writes a marker. - Start serve:
swamp serve --no-schedule --shutdown-drain-timeout 3s. - Run
swamp workflow run <wf> --server <url> --detach. - While
workruns, cancel it withswamp workflow cancel <run> --server <url>. The reply iscancellation_requested. - About 4 s into A's cleanup, send serve SIGTERM.
Expected
A's cleanup finishes, then job B starts and finishes, the same as with no SIGTERM. The run ends cancelled with B succeeded. If the cleanup grace ran out, serve would log Saved cancelled workflow run …: shutdown cut off its cleanup.
Actual
- A's cleanup finishes and serve exits 0 about 3.4 s after SIGTERM, which is as soon as A's cleanup ends.
- Job B never starts: there is no started marker.
- History shows
Bfailed andB/on-failurefailed: cancelled. The run endscancelled. - The roughly 20 s cleanup grace goes unused, and no cut-off is logged.
Control, with the same cancel and no SIGTERM: B starts and succeeds.
Root cause (from source)
The SIGTERM handover added by swamp-club#4737 also reaches the cleanup jobs of a run the user cancelled before the shutdown.
src/domain/workflows/execution_service.tsaround line 8718:cleanupStepOpts = { ...stepOpts, signal: cleanupSignal, cleanupJobLevel: true }keepsstepOpts.handover.markStarted(around lines 3388-3398) returns without starting the job whenhandedOver(handover).runShutdownDraincorrectly treats the run as cancelled-before-shutdown and would waitstopGraceFor, but the run has already settled by then.
A user-cancelled run's cleanup jobs should ignore the shutdown handover and run under the cleanup-grace signal, as they do on a cancel with no shutdown.
Environment
- swamp built from main
8b8e14c6, Linux. - Found by the swamp-uat test
tests/cli/serve/none/shutdown_cleanup_test.ts: "swamp serve SIGTERM lets a run cancelled before the shutdown finish its cleanup, and leaves it cancelled, not interrupted" (swamp-uat#976).
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.