Relationships
#4655 serve: a rolling restart's boot sweep expires the leases, dispatches and workers the old process is still using
Opened by stack72 · 11/11/2026· Shipped 11/12/2026
Found while designing part c of #4439 (runs survive a restart).
What happens
During a rolling restart with --reuse-port (#4566), the new serve process starts while the old one is still draining. At boot the new process runs sweepStaleRecords (src/serve/boot_reconciliation.ts, called from settleLeftRuns in src/cli/commands/serve.ts). The sweep filters only on state, never on owner:
- every
activestep lease is expired, with error "orchestrator restart"; - every
waitingpending dispatch is markedorphaned; - every worker record is marked
disconnected, withactiveDispatchIds: [].
That includes the leases, dispatches and workers the old process is still using. Its in-memory dispatches keep running, but the durable records now say they expired or were orphaned, and that their workers are gone. Anything that reads those records then sees the wrong state while the old process drains: the worker pool reader for fan-out lanes, the dashboard and health worker views, and lease checks on the old process's dispatches. A worker result reaching the old process can land on a lease that has already been expired.
The same sweep runs on promotion in team mode (onLead), where the previous leader may still be finishing its own work.
Expected
The sweep settles only records whose owner is gone. A record owned by a process that is still alive is left alone until that process exits or is judged dead. Queued runs already work this way: pending runs carry PendingRunOwner {instanceId, pid, host}, and replay defers a run whose owner is alive. Leases, pending dispatches and worker records need the same ownership check: record the owning orchestrator on each, and skip records whose owner is alive.
Test
Two serve processes on one repo, the second started while the first still holds an active lease and a waiting dispatch. The second process's boot leaves both records as they were. Once the first exits, the next sweep, or a later boot, settles them.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.