Relationships
#4762 serve: a run left RUNNING after a restart is never seen as stale when the new serve reuses the old PID (containers)
Opened by randybias · 11/14/2026· Shipped 11/16/2026
What happens
- A workflow run is in flight on
swamp serve. The serve exits during its shutdown drain (SIGTERM, drain timeout, then exit) without recording the run as interrupted. - The serve runs in a container with an init process as PID 1, so the serve itself is PID 2. A supervisor (systemd
Restart=always) starts it again, and the new serve is PID 2 again. - At boot the new serve logs:
Skipping workflow run "<id>" … — tracker reports still running. The run stays RUNNING. swamp run doctorlists the run as stale (heartbeat 2 minutes old, PID 2), but the run stays RUNNING.swamp workflow cancel --run <id>refuses: "is recorded as running and was not cancelled: the process running it on …".swamp workflow recover <wf> --run <id>says "Interrupted run not found".- What finally cleared it:
swamp run doctor --fix, and then the next serve start. That serve logged the run as "interrupted by server_crash" and continued it.
Expected
A run whose serve process has exited should be found stale, even if a new process later holds the same PID on the
same host. Heartbeat age, a process start time, or a per-boot id would tell the two apart. cancel should be able to
end such a run.
Why it matters
Containerised serves with an init process (tini, catatonit, podman's --init) always run the serve as the same PID.
Any missed interrupted mark then leaves a run that nothing short of doctor --fix and a restart can end.
Shortest reproduction
Run swamp serve in a container with tini as PID 1, under a supervisor that restarts it. Start a workflow through
the serve whose step sleeps without honouring the method's abort signal. Send the serve SIGTERM. After the restart,
the run stays RUNNING as described.
Versions
swamp 20261114.115103.0-sha.f37e02ff, Linux, rootless podman 4.9.3, Ubuntu 24.04 host, debian:12-slim image.
Related, with a different cause: #3355 (a run reaped by a peer) and #3962 (a v3 datastore). Here it's one serve, one host, and the same PID after the restart.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 11/16/2026, 8:52:02 PM
Thanks @randybias for reporting this! We shipped: The run tracker judges a row's owner by pid alone and treats its own pid as alive, so a serve restarted under the same pid (PID 2 under tini, catatonit, or podman --init) keeps the dead serve's row running and never reaps or lets cancel end the run. Record each tracker row owner's process start time (schema v10) and judge a row's owner gone when its pid is alive under another start time, or, for a row with no start time, when it is this process's pid under another instance id. Apply that one judgment, carrying the caller's instance id, in the boot and periodic reaps, run doctor, recover, and serve's cancel decider, so a restarted serve settles the run at boot and continues it.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.