Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeshammz

Relationships

#3059 run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Opened by hammz · 10/6/2026· Shipped 10/6/2026

Found while triaging #3051. Separate from it, and from #2518, which fixed the same mistake in the cancel path only.

Problem

handleRunDoctor in src/serve/handlers/admin_handlers.ts treats a running workflow run as orphaned when its instanceId is not this instance and the control-plane store holds no heartbeat for that instance. Serve only writes instance heartbeats when the datastore has a control-plane capability (InstanceHeartbeatService is created under hasRemoteControlPlane in src/cli/commands/serve.ts). Without one, serve still builds a local FileSystemControlPlaneStore, so the handler's guard (a control-plane store and an instance id exist) passes, no heartbeat is ever found, and every run owned by any other serve instance looks orphaned. With --fix the handler interrupts it.

Reproduction

Build of origin/main at fcfe2326. Scratch repo with the default datastore, one workflow whose only step sleeps 60 seconds.

  1. Start two serve processes A and B on the same repo (both log: Control-plane store: local filesystem fallback).
  2. Start the workflow through B. Its record shows status running and B's instanceId.
  3. swamp run doctor --server A --json reports orphanedWorkflowRuns 1 while B is alive and the run is in progress. The same response lists the run under activeRuns with B's live pid.
  4. swamp run doctor --fix --server A --json reports orphanedReaped 1, and the record is now status interrupted although B is still executing it.

Expected: orphanedWorkflowRuns 0. A missing heartbeat means nothing where heartbeats are not being recorded.

Pointer

ownerGoneDecider in src/serve/suspended_run_cancel.ts already handles this since #2518: it consults the run tracker row for a local owner first, and it only trusts a missing heartbeat when this instance's own heartbeat is present in the store. The doctor handler could use the same decision instead of its own heartbeat lookup. The boot reaper is not affected in the same way, since it only reaps runs of instances it claimed.

Scratch repo and script: /tmp/swamp-repro-issue-3051/hb and /tmp/swamp-repro-issue-3051/hb.sh on the triage machine.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

10/6/2026, 5:11:01 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
hammz assigned hammz10/6/2026, 4:15:10 PM

Sign in to post a ripple.