Relationships
↔ sibling #4656#4660 serve: shutdown does not wait for or cancel the own-cron replay that follows a scheduler turn-on
Opened by stack72 · 11/11/2026· Shipped 11/12/2026
Follow-up to #4652 (PR #3749): the CI adversarial review raised a medium that this fix did not cover.
What happens
#4652 made ScheduledExecutionLifecycle.drain() and stop() (src/serve/scheduled_execution_lifecycle.ts) wait on #starting, which covers only the start phase of turnOn: build, deps.start, publish. The replay that follows is not waited on:
turnOncalls#replayThenCatchUp(service, createdBefore)after publishing, when it leads.atLeadcalls#replayThenCatchUp(service, cutoff)for a turn-on that did not lead. This window has existed since #4633.#replayThenCatchUpawaitsdeps.replayOwnCron(...). The CLI wires that toreplayPendingRuns({ ownCronOnly: true, ... })insrc/serve/boot_reconciliation.ts, which readsrunTracker.findAllPendingRuns()andcontrolPlaneStore.list("pending-runs/")plus onegetper key (remote on S3/GCS), then callsscheduledExecution.enqueueForReplay(...)per own cron entry.
If SIGTERM lands during the replay, shutdown drains and stops the scheduler and later closes the stores while the replay is still reading them.
Current effect (to verify)
- A read on a closed store throws, is caught, and logs a warning ("Replaying pending scheduled runs failed; they stay pending").
enqueueForReplayreturns early once the service is draining, so the pending entry stays for the next boot. No run is lost.replayPendingRunsstill counts the entry and logs "Replayed pending cron run for ...", which is misleading.- Catch-up is skipped through the
#closedcheck.
Expected
Shutdown either aborts the replay or waits for it within a bound. A slow or unreachable bucket must not hold shutdown, so the wait must not be unbounded. Preferred: thread an AbortSignal through replayOwnCron and replayPendingRuns, so stop() aborts the control-plane reads. Otherwise stop() waits for the replay within a bound (the shutdown deadline, or a fixed settle bound like FIRE_SETTLE_TIMEOUT_MS). Optionally, an enqueue that the scheduler dropped is neither counted nor logged as replayed.
Test
In src/serve/scheduled_execution_lifecycle_test.ts, with replayOwnCron held: stop() aborts or waits for the replay and resolves within the bound, and no catch-up runs after it, on both the turnOn and atLead paths. A replay that never settles does not hold stop() past the bound.
Out of scope
Overlapping reloads (performServeReload already refuses a second reload in progress). Deleting stale remote-only own-cron entries (#4652 deliberately skips them).
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.