#3739 serve: a run orphaned by a container restart can't be cancelled — the restarted serve reuses the dead owner's pid, so cancel reports the owner still alive

Summary

A serve run orphaned by a fast container restart can't be cancelled: workflow cancel --server refuses it as "the process running it on the serve host is still alive". The live process it means is the restarted serve itself. Serve runs as pid 1 again (or another low, deterministic pid) under the same hostname, so the dead owner's tracker row names the new serve's pid. On a filesystem datastore nothing reaps the row while that serve keeps running.

Seen in the field:

{
  "error": "Run 65eb7534-c40b-47e9-a287-41bc3edbbbda is recorded as running and was not cancelled: the process running it on the serve host is still alive. Stop that process, then cancel the run again"
}

Reproduction (verified, swamp 20261021.155838.0-sha.f979f7c1, filesystem datastore, auth-mode none)

unshare --user --map-root-user --pid --fork --mount-proc stands in for a container: serve is pid 1, hostname unchanged.

1. Repo with a command/shell behavior sleeper and a workflow sleepwf whose step runs execute with run: sleep 600.
2. Start serve as pid 1: unshare --user --map-root-user --pid --fork --mount-proc sh -c 'exec swamp serve --port 9123'
3. swamp workflow run sleepwf --server ws://127.0.0.1:9123 --detach → run 203d9e33…. Tracker row: pid=1, hostname=mimir, instance_id=b71f8d06…, status=running.
4. SIGKILL serve (as in a container crash or OOM kill). The step's sleep dies with it.
5. Within 90 s, start serve again the same way: pid 1, same hostname, new instance id.
6. swamp workflow cancel sleepwf --run 203d9e33… --server ws://127.0.0.1:9123 → the error above. The run record and both tracker rows (workflow + model_method) stay running.
7. Retried after the row's tracker heartbeat was 100 s old: same refusal. It stays stuck while this serve runs.
8. Workaround: restart serve once more, at least 90 s after the crash. Boot reapStaleRuns now finds the row stale and from another instance and interrupts it ("Reaped stale workflow run…", "1 workflow run(s) interrupted by crash"). swamp workflow recover can then take over.

Cause

• ownerGoneDecider (src/serve/suspended_run_cancel.ts:121-130) judges a row with this host's hostname by its pid alone. It never checks whether that pid is serve's own (Deno.pid) for a run not in serve's activeRunRegistry, or whether the row's instanceId differs from this instance's. With pid reuse, isProcessGone(1) is false → OWNER_PROCESS_ALIVE.
• The boot pass does not catch it: reapStaleRuns(DEFAULT_STALE_TTL_MS) only considers rows whose tracker heartbeat is older than 90 s, and a restart policy brings serve back sooner. reapOrphanedWorkflowRuns then sees a running tracker row and leaves the run.
• Without a remote control plane there is no periodic reconciliation (reconcileRemoteInterruptedRuns runs only when hasRemoteControlPlane). Nothing looks at the row again until the next boot.
• By contrast, goneInstanceIds (src/domain/workflows/orphaned_run_reaper.ts:224) already handles this case: a row carrying the caller's own pid is judged on heartbeat age, not pid liveness ("A restarted container reuses its pid…").
• The existing test ownerGoneDecider: a live pid in the tracker row keeps the owner, whatever the control plane says uses Deno.pid under old-instance as its "live owner", so it locks in this behavior.

Expected

A run whose tracker row belongs to a dead serve instance can be cancelled from the restarted instance on the same host, even when the pid was reused. At minimum, a row whose pid is serve's own and whose run is not in serve's registry should not count as a live owner. Ideally the boot pass also interrupts such rows rather than waiting for a later restart.

Related

• swamp-club#3564 (item 5): pid reuse with a stable hostname makes a dead v3 writer look alive. Same root weakness: rows carry only host + pid. processHostIdentity() (hostname#pidns) exists for this.
• swamp-club#2518: cancelling a running run whose serve process is gone.

Environment

• swamp 20261021.155838.0-sha.f979f7c1 (no changes to the involved files through main 15d680c4e)
• Linux, filesystem datastore, serve as pid 1 in a pid namespace (stands in for a Docker/k8s container with a stable hostname)