Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
AssigneesNone

Relationships

↔ sibling #4656

#4660 serve: shutdown does not wait for or cancel the own-cron replay that follows a scheduler turn-on

Opened by stack72 · 11/11/2026· Shipped 11/12/2026

Follow-up to #4652 (PR #3749): the CI adversarial review raised a medium that this fix did not cover.

What happens

#4652 made ScheduledExecutionLifecycle.drain() and stop() (src/serve/scheduled_execution_lifecycle.ts) wait on #starting, which covers only the start phase of turnOn: build, deps.start, publish. The replay that follows is not waited on:

  • turnOn calls #replayThenCatchUp(service, createdBefore) after publishing, when it leads.
  • atLead calls #replayThenCatchUp(service, cutoff) for a turn-on that did not lead. This window has existed since #4633.
  • #replayThenCatchUp awaits deps.replayOwnCron(...). The CLI wires that to replayPendingRuns({ ownCronOnly: true, ... }) in src/serve/boot_reconciliation.ts, which reads runTracker.findAllPendingRuns() and controlPlaneStore.list("pending-runs/") plus one get per key (remote on S3/GCS), then calls scheduledExecution.enqueueForReplay(...) per own cron entry.

If SIGTERM lands during the replay, shutdown drains and stops the scheduler and later closes the stores while the replay is still reading them.

Current effect (to verify)

  • A read on a closed store throws, is caught, and logs a warning ("Replaying pending scheduled runs failed; they stay pending").
  • enqueueForReplay returns early once the service is draining, so the pending entry stays for the next boot. No run is lost.
  • replayPendingRuns still counts the entry and logs "Replayed pending cron run for ...", which is misleading.
  • Catch-up is skipped through the #closed check.

Expected

Shutdown either aborts the replay or waits for it within a bound. A slow or unreachable bucket must not hold shutdown, so the wait must not be unbounded. Preferred: thread an AbortSignal through replayOwnCron and replayPendingRuns, so stop() aborts the control-plane reads. Otherwise stop() waits for the replay within a bound (the shutdown deadline, or a fixed settle bound like FIRE_SETTLE_TIMEOUT_MS). Optionally, an enqueue that the scheduler dropped is neither counted nor logged as replayed.

Test

In src/serve/scheduled_execution_lifecycle_test.ts, with replayOwnCron held: stop() aborts or waits for the replay and resolves within the bound, and no catch-up runs after it, on both the turnOn and atLead paths. A replay that never settles does not hold stop() past the bound.

Out of scope

Overlapping reloads (performServeReload already refuses a second reload in progress). Deleting stale remote-only own-cron entries (#4652 deliberately skips them).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPEDLINKED+ 3 MOREPR_LINKED+ 1 MORESESSION_SUMMARIZED

Shipped

11/12/2026, 9:16:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 linked sibling of #465611/11/2026, 7:11:32 PM

Sign in to post a ripple.