Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4655 serve: a rolling restart's boot sweep expires the leases, dispatches and workers the old process is still using

Opened by stack72 · 11/11/2026· Shipped 11/12/2026

Found while designing part c of #4439 (runs survive a restart).

What happens

During a rolling restart with --reuse-port (#4566), the new serve process starts while the old one is still draining. At boot the new process runs sweepStaleRecords (src/serve/boot_reconciliation.ts, called from settleLeftRuns in src/cli/commands/serve.ts). The sweep filters only on state, never on owner:

  • every active step lease is expired, with error "orchestrator restart";
  • every waiting pending dispatch is marked orphaned;
  • every worker record is marked disconnected, with activeDispatchIds: [].

That includes the leases, dispatches and workers the old process is still using. Its in-memory dispatches keep running, but the durable records now say they expired or were orphaned, and that their workers are gone. Anything that reads those records then sees the wrong state while the old process drains: the worker pool reader for fan-out lanes, the dashboard and health worker views, and lease checks on the old process's dispatches. A worker result reaching the old process can land on a lease that has already been expired.

The same sweep runs on promotion in team mode (onLead), where the previous leader may still be finishing its own work.

Expected

The sweep settles only records whose owner is gone. A record owned by a process that is still alive is left alone until that process exits or is judged dead. Queued runs already work this way: pending runs carry PendingRunOwner {instanceId, pid, host}, and replay defers a run whose owner is alive. Leases, pending dispatches and worker records need the same ownership check: record the owning orchestrator on each, and skip records whose owner is alive.

Test

Two serve processes on one repo, the second started while the first still holds an active lease and a waiting dispatch. The second process's boot leaves both records as they were. Once the first exits, the next sweep, or a later boot, settles them.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/12/2026, 5:35:41 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/11/2026, 6:07:40 PM

Sign in to post a ripple.