Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#3472 serve shutdown waits for an unsignalled run index rebuild: the continuation sweep's boot pass and request-led rebuilds cannot be stopped by SIGTERM

Opened by hammz · 10/14/2026

Summary

When a workflow's run index is missing or stale at serve start, SIGTERM cannot stop the index rebuild or repair in flight, and shutdown waits for it. Exit takes as long as the rebuild (about 7 s for 30k runs, 17-19 s when two readers rebuild one after the other on older builds).

This predates swamp-club#3443 (#3105): measured on 3999f85e and on e99b04d4. Found while triaging swamp-club#3468, which fixes only the run index warm-up's share of the delay.

Cause

  • ContinuationSweepService's boot pass calls findSummariesByStatus(wf, suspended), which reaches getValidatedIndex / repairIndex with no AbortSignal. Shutdown awaits continuationSweepService.dispose() (src/cli/commands/serve.ts, after the drain), so it waits for that rebuild.
  • workflow.run.summary and run search (src/serve/handlers/workflow_handlers.ts, the runIndexSnapshot calls near lines 1292 and 1405) also rebuild with no signal. Other readers of the same index wait behind them in withIndexQueue, including the sweep.

Repro

  1. A workflow with 30k run records and an intact .runs-index.json.
  2. Start swamp serve. Delete the index right after it starts listening.
  3. Optionally send a workflow.run.summary for that workflow.
  4. Send SIGTERM 2.5 s later.

Measured (SWAMP_NO_TELEMETRY=1, 30k runs):

  • Sweep alone (no request): OLD 7.0-7.5 s, NEW 7.2-7.3 s.
  • With the request: OLD 7.0 s or 17-19 s, NEW 7-10 s.
  • Index left intact: both 0.03 s.

Suggested direction

  • Give the sweep's boot pass a signal that dispose() aborts, and pass it into the index read so a rebuild stops between records and writes no index.
  • Consider whether request handlers should get a signal tied to server shutdown.
  • A reader waiting in withIndexQueue should also be able to stop waiting when its signal aborts.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/14/2026, 7:24:37 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.