#3355 serve: a run whose crashed owner was reaped by a peer stays running on the restarted owner

Severity: medium — the run record and run history report running for a run whose process is dead, so anything polling the status waits on it; it is not permanent (a later serve boot at least 90 s after the crash, or swamp run doctor --fix, clears it) and no data is lost.

Affects: found on swamp 20261009.215038.0-sha.986f9022; the UAT test below still fails the same way on 20261013.025835.0-sha.1b6f0c59 (2026-10-13). The hand reproduction and the source references were taken at 986f9022 and not repeated on the newer build.

Related: #2243 (shipped) covered runs left running after a clean rolling restart of a single replica. This is a different path: a SIGKILLed owner whose stale heartbeat a peer claims first.

Summary

Two swamp serve instances share an S3 datastore. Instance A is SIGKILLed while driving a workflow run. Instance B notices A's stale heartbeat, claims it, deletes A's active-runs/ record and heartbeat, and interrupts nothing, because B has no tracker row or run record for A's run. When A is then started again on its own repo, its boot skips the run with tracker reports still running, and swamp workflow history get <runId> and swamp run history keep answering running.

Nothing in the running serve process corrects this afterwards. In the reproduction the run was still running 100 s after the restart. It became interrupted only when A was restarted a second time, more than 90 s after the crash.

Without the peer (same steps, B never started) the restarted A interrupts the run during boot, so the peer's cleanup is what strands it.

With default settings (--stale-ttl 90 s, --reconciliation-interval 60 s) the same sequence needs the crashed instance to be back up before a peer's reconciliation tick claims its stale heartbeat, and the peer's tick to win the claim rather than the restarted instance's own tick. This ordering was only reproduced with the short intervals below.

Steps to reproduce

Needs Docker, jq and curl. Any S3-compatible endpoint works; this uses a throwaway MinIO.

export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin AWS_REGION=us-east-1
docker run -d --rm --name repro-minio -p 127.0.0.1:19909:9000 \
  quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
sleep 3 && docker exec repro-minio mkdir -p /data/repro

W=$(mktemp -d); NS=ns-$(date +%s)
for i in a b; do
  mkdir -p $W/$i/models/command/shell $W/$i/workflows && cd $W/$i
  swamp init
  swamp datastore setup extension @swamp/s3-datastore --json --config \
    "{\"bucket\":\"repro\",\"region\":\"us-east-1\",\"prefix\":\"p1\",\"namespace\":\"$NS\",\"endpoint\":\"http://127.0.0.1:19909\",\"forcePathStyle\":true}"
  cat > models/command/shell/d29f99fc-ef5f-5338-bef8-9273bfd4e7b7.yaml <<'Y'
type: command/shell
typeVersion: 2026.02.09.1
id: d29f99fc-ef5f-5338-bef8-9273bfd4e7b7
name: shell-sleep
version: 1
tags: {}
globalArguments:
  run: "sleep 30"
methods:
  execute:
    arguments:
      run: "sleep 30"
Y
  cat > workflows/workflow-2ac4237a-1ecd-4f91-adc4-476489876501.yaml <<'Y'
id: 2ac4237a-1ecd-4f91-adc4-476489876501
name: sleep-step
tags: {}
jobs:
  - name: sleep-job
    steps:
      - name: sleep-run
        task:
          type: model_method
          modelIdOrName: shell-sleep
          methodName: execute
        dependsOn: []
        weight: 0
    dependsOn: []
    weight: 0
version: 1
Y
done

SERVE="--auth-mode none --no-schedule --heartbeat-interval 1s --stale-ttl 2s --reconciliation-interval 2s"
A=http://127.0.0.1:19911; B=http://127.0.0.1:19912
ready() { until curl -sf $1/ready >/dev/null; do sleep 0.5; done; }

(cd $W/a && exec swamp serve --port 19911 $SERVE > $W/a1.log 2>&1) & PID_A=$!; ready $A
(cd $W/b && exec swamp serve --port 19912 $SERVE > $W/b1.log 2>&1) & PID_B=$!; ready $B

cd $W/a
swamp workflow run sleep-step --server $A > $W/client.log 2>&1 &
until grep -q "step sleep-run" $W/client.log; do sleep 0.5; done
RUN=$(swamp run history --json --server $A 2>/dev/null | grep '^{' \
  | jq -r '.runs[] | select(.runKind=="workflow" and .status=="running") | .id')

kill -9 $PID_A
until grep -q "Cleaned up 1 stale active-run records" $W/b1.log; do sleep 0.5; done

(cd $W/a && exec swamp serve --port 19911 $SERVE > $W/a2.log 2>&1) & PID_A=$!; ready $A
grep "Skipping workflow run" $W/a2.log
sleep 5
swamp workflow history get $RUN --json --server $A | grep '^  "status"'   # "running"

# cleanup
kill $PID_A $PID_B; docker stop repro-minio

Control: skip starting instance B. The restarted A then logs Reaped YAML workflow run "<runId>" from dead remote instance "<old A id>" and reports interrupted.

Expected behaviour

After the owning serve process has been killed and the instance is back up, the run is interrupted (in-flight steps unknown), whether or not a peer reaped the dead instance first.

• design/primitives/workflows.md:3146 ("Recovery"): "When a serve instance crashes during a run, or a run's owning process dies, the run is marked interrupted and its in-flight steps unknown."
• design/primitives/serve.md:924 (Boot step 4): "Reaps runs whose owning PID or heartbeat is gone". In the stuck state both are gone: the pid is dead and the heartbeat has been deleted.
• design/primitives/serve.md:1018-1021: "the next instance's boot reconciliation interrupts runs from foreign instances whose heartbeat is gone. After a crash, the reconciliation loop handles the dead instance once --stale-ttl passes."
• design/primitives/serve.md:932-936 ("Steady state") says the reconciling peer "marks that peer's running tracker rows interrupted ..., interrupts its YAML workflow-run records and deletes its active-runs/ records". Here the peer did only the last of the three.
• The same crash without a peer ends interrupted within seconds of the restart (control above).

Actual behaviour

Output of the reproduction (timestamps local; instance and run ids vary):

A instanceId=873b828d-5765-42b4-bda7-4e5d9c7287fd
RUN=6134d290-0341-4bbf-a77e-2f2c0cbdb6db
09:33:01 kill -9 A
# instance B log:
 system │ Cleaned up 1 stale active-run records for instance "873b828d-5765-42b4-bda7-4e5d9c7287fd"
09:33:05 A restarted (new instanceId=014425f5-28f7-432f-abcb-1790ef3ecaba)
# restarted A boot log:
 system │ Skipping workflow run "6134d290-0341-4bbf-a77e-2f2c0cbdb6db" (workflow: "sleep-step") — tracker reports still running

swamp workflow history get 6134d290-... --json --server http://127.0.0.1:19911 exits 0 with "status": "running" (job and step also running) at 5 s, 35 s, 65 s and 100 s after the restart. swamp run history --json --server <A> shows the same row:

{"id":"6134d290-0341-4bbf-a77e-2f2c0cbdb6db","runKind":"workflow","status":"running","stale":false}   # 5 s to 65 s
{"id":"6134d290-0341-4bbf-a77e-2f2c0cbdb6db","runKind":"workflow","status":"running","stale":true}    # 100 s

The same history get against B exits 1: Server reported workflow_history_get_failed: No workflow run or workflow found: 6134d290-....

What clears it:

• Restarting A again 111 s after the crash. Its boot logs Reaped stale "workflow" run "6134d290-..." ("sleep-step"), then Reaping orphaned workflow run ... (reason: "tracker confirmed stale"), and the run reports "status": "interrupted", interrupted: server_crash.
• swamp run doctor run in A's repo lists it under "1 stale run(s) detected" and "1 orphaned workflow run(s) whose owner is gone" and offers --fix (the fix itself was not exercised).

Root cause

All paths at 986f9022.

1. src/serve/boot_reconciliation.ts:541 reconcileRemoteInterruptedRuns, run by the peer. After claiming the stale instance it interrupts only rows in its own run tracker (:590, runTracker.findAllRunning()) and run records in its own local cache (:627, workflowRunRepo.findGlobalByStatus("running")). The run tracker is local SQLite and a run record is not pushed until the run ends, so for a run that crashed mid-step the peer finds neither. It still deletes the dead instance's active-run records (:606) and heartbeat (:651), which removes the only signal the owner's own boot would have acted on.
2. src/cli/commands/serve.ts:4174 gives every serve process a new instanceId. On the restarted owner:
   • reapDeadProcessRuns(instanceId) (serve.ts:4349, src/infrastructure/persistence/run_tracker_store.ts:433) filters with ActiveRun.isLocalTo (src/domain/models/active_run.ts:117), which compares instance ids when both sides have one. The old row carries the previous instance id, so it is not treated as local and its dead pid is never checked. The comment at serve.ts:4401-4405 notes exactly this for method runs ("judged on host and pid, not instance id: this serve's instance id is new") and handles it there; the workflow row gets no such treatment.
   • reapStaleRuns(DEFAULT_STALE_TTL_MS, instanceId) (serve.ts:4340) uses the fixed 90 s default (src/serve/instance_heartbeat.ts:27), not --stale-ttl, so a row younger than 90 s is left running. It runs only at boot, which is why a later restart clears the run.
   • reapOrphanedWorkflowRuns (src/domain/workflows/orphaned_run_reaper.ts:66) skips any run whose tracker row is running. Its check for a foreign instance with no heartbeat (:96) is reached only when there is no tracker row.
   • The boot pass of reconcileRemoteInterruptedRuns (serve.ts:6936) finds no stale heartbeat, because the peer deleted it.

The peer-side cleanup of active-runs/ records was added in c39cd2d0 (#2049, 2026-08-03); whether an earlier revision behaved differently was not checked.

How the UAT found it

Test: tests/cli/serve/oauth/reconciliation_test.ts, "swamp serve reaps interrupted runs from a dead peer".

Before the audit the test was ignore: true (marked TODO(lab-bug)), so it never ran. Its body asserted something different: that the surviving peer itself answers workflow history get <runId> with interrupted. That fails for another reason (the peer has no record of the run and answers "No workflow run or workflow found"), and the design docs disagree on whether a peer should hold that record, so the audit did not keep that assertion.

The audit (commit 6c14b7c on the swamp-uat branch test-investigation, not yet on swamp-uat main) un-ignored the test and changed it to: read the owner's instance id from /ready, SIGKILL the owner mid-run, wait for the survivor to log Cleaned up 1 stale active-run records for instance "<owner id>", start the owner again on its own repo, and require swamp workflow history get <runId> --json --server <owner> to report interrupted within 30 s. It fails with the last answer still "status": "running", exit 0, after 36 attempts.

Run it (needs an aws CLI on PATH and Docker for the S3 emulator the helpers start):

deno test --allow-read --allow-write --allow-env --allow-run --allow-net \
  --filter 'reaps interrupted runs' tests/cli/serve/oauth/reconciliation_test.ts

The sibling test in the same file, "swamp serve rejects runs over the concurrency cap and frees a crashed run's slot after a restart", covers the single-instance restart.