Relationships
#4652 serve: close three timing windows in the scheduler reload lifecycle (boot reload, shutdown during turn-on, stale remote own-run replay)
Opened by stack72 · 11/11/2026· Shipped 11/11/2026
Follow-up to swamp-club#4633 (PR #3746). The CI adversarial review on that PR passed, but noted three timing windows in the new scheduler reload code. Each is small to close.
1. A reload during boot can start scheduling before the boot replay
The SIGHUP listener is registered before boot replays pending runs, and before replayOwnPendingCron is assigned in src/cli/commands/serve.ts.
Scenario:
- Boot with
schedule: false. - A SIGHUP reload sets
schedule: truewhile boot is between those two points.
What goes wrong:
ScheduledExecutionLifecycle.turnOn(src/serve/scheduled_execution_lifecycle.ts) seesleads()as true.- The own-run replay is still the stub, which returns 0.
- So
turnOncallsstartCatchUp()straight away. Catch-up runs then queue ahead of the boot replay of pending runs, which breaks the invariant "after the replay, so a replayed run queues ahead of a catch-up". - Boot then calls
startCatchUp()a second time.
A catch-up slot may also overlap a replayed pending run for the same slot.
Fix: the schedule follow-up in timedServiceFollowUps throws "serve is still starting" until boot's pending replay has run, as the late-built timed services already do. The setting is then reported as not applied and retried on the next reload.
2. Shutdown can miss a scheduler a reload is still starting
turnOn awaits deps.start(service) (workflowRepo.findAll() and service.start()) before it sets #current. If SIGTERM arrives in that window:
drain()andstop()see no current and no retiring scheduler, so they return at once.- Shutdown goes on to close stores while
service.start()is still reading.
The #closed check after the await does stop the service afterwards, so nothing is lost; the effect is error noise from reads against closed stores.
Fix: hold the scheduler being started in a field that drain() and stop() include, or have them await the in-flight turnOn.
3. The own-run replay can re-run a run whose control-plane delete failed
replayPendingRuns with ownCronOnly (src/serve/boot_reconciliation.ts) merges remote pending-runs/ entries that are missing locally. It skips the "already owned by this process" check.
Two ways a stale remote entry owned by this instance gets enqueued again on a turn-on:
- the serve pending-run hook (
src/cli/commands/serve.ts) deletes the local entry, then logs and swallows a failed control-plane delete; - the delete is still in flight when the turn-on lists remote entries.
Before #4633, only a restart replayed such an entry.
Fix: in ownCronOnly mode, skip remote entries that are not in the local run tracker. Every run this process left pending is in its local tracker, because the enqueue writes there first.
Tests
- Item 1: a reload that turns scheduling on before boot's replay is refused with the starting reason, and it applies on the next reload.
- Item 2: a
turnOnwhose start is still pending whendrain/stoprun is waited for and stopped (lifecycle unit test with a held start). - Item 3: a remote-only own cron entry is not replayed by
ownCronOnly(boot_reconciliation_test.ts).
Red-check each test against the current code.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.