Relationships
#4763 serve: during the shutdown drain, an aborted scheduled run keeps executing its remaining steps
Opened by randybias · 11/14/2026· Shipped 11/16/2026
What happens
swamp servereceives SIGTERM while a scheduled run is in flight. It drains for up to--shutdown-drain-timeout(30s).- At the deadline it logs
Aborting in-flight scheduled run for workflow "<id>", and the step's AbortSignal fires. - The run does not stop. Its failure-path steps run next, inside the draining serve: steps that depend on the aborted step having failed, and the jobs that depend on that job failing.
- Only about 30 seconds later does serve log
Interrupted workflow run "<id>" (server shutdown)and exit. - For a run started through the API (a person's
workflow run --server), the same abort is followed within about 50 ms by the interrupted mark, and no further step runs.
Expected
An aborted run should stop at the step that was aborted, whether a schedule or a person started it. It should then be marked interrupted, so the next serve can continue it under the usual guard rules.
Why it matters
A workflow that restarts its own serve (for example, to load a renewed TLS certificate) runs its verify and rollback steps against the old, draining process when a schedule started it. The rollback acts on a wrong reading. The same workflow behaves correctly when a person starts it.
Shortest reproduction
A workflow with step A, which sends SIGTERM to the serve and settles when the method's AbortSignal fires, and job B,
which depends on A's job failing. Run it once from a schedule and once with workflow run --server. On the
scheduled run, B executes before the serve exits. On the person's run, it does not.
Versions
swamp 20261114.115103.0-sha.f37e02ff, Linux, rootless podman 4.9.3.
Shipped
Click a lifecycle step above to view its details.
zack commented 11/16/2026, 6:31:35 PM
Thanks @randybias for reporting this! We shipped: A serve shutdown abort is treated as a cancellation, so the workflow engine runs failure-path (cleanup) jobs and steps under a fresh 30s grace inside the draining serve, for API, cron and webhook runs alike; scheduled and webhook runs additionally wait unboundedly in stop(), so the interrupt lands up to 30s later. Fix: a typed RunInterruption abort reason that the engine reads as an interruption at all three cleanup sites (no cleanup starts; unstarted work is settled by abort and reopened on resume; in-flight steps end unknown), used by all three shutdown abort sites, and scheduled/webhook stop() (and retiring schedulers) bounded by the same 5s abort grace the registry gets, with all three aborted concurrently.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.