Relationships
#3059 run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded
Opened by hammz · 10/6/2026· Shipped 10/6/2026
Found while triaging #3051. Separate from it, and from #2518, which fixed the same mistake in the cancel path only.
Problem
handleRunDoctor in src/serve/handlers/admin_handlers.ts treats a running workflow run as orphaned when its instanceId is not this instance and the control-plane store holds no heartbeat for that instance. Serve only writes instance heartbeats when the datastore has a control-plane capability (InstanceHeartbeatService is created under hasRemoteControlPlane in src/cli/commands/serve.ts). Without one, serve still builds a local FileSystemControlPlaneStore, so the handler's guard (a control-plane store and an instance id exist) passes, no heartbeat is ever found, and every run owned by any other serve instance looks orphaned. With --fix the handler interrupts it.
Reproduction
Build of origin/main at fcfe2326. Scratch repo with the default datastore, one workflow whose only step sleeps 60 seconds.
- Start two serve processes A and B on the same repo (both log: Control-plane store: local filesystem fallback).
- Start the workflow through B. Its record shows status running and B's instanceId.
- swamp run doctor --server A --json reports orphanedWorkflowRuns 1 while B is alive and the run is in progress. The same response lists the run under activeRuns with B's live pid.
- swamp run doctor --fix --server A --json reports orphanedReaped 1, and the record is now status interrupted although B is still executing it.
Expected: orphanedWorkflowRuns 0. A missing heartbeat means nothing where heartbeats are not being recorded.
Pointer
ownerGoneDecider in src/serve/suspended_run_cancel.ts already handles this since #2518: it consults the run tracker row for a local owner first, and it only trusts a missing heartbeat when this instance's own heartbeat is present in the store. The doctor handler could use the same decision instead of its own heartbeat lookup. The boot reaper is not affected in the same way, since it only reaps runs of instances it claimed.
Scratch repo and script: /tmp/swamp-repro-issue-3051/hb and /tmp/swamp-repro-issue-3051/hb.sh on the triage machine.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.