#4286 serve shutdown waits minutes after the drain ends, beyond the documented drain + ~25 s bound

Summary

swamp serve documents shutdown as bounded by --shutdown-drain-timeout plus about 25 s ("Keep it below the pod's terminationGracePeriodSeconds minus about 25s"). On one serve, with the default 30 s drain, the drain finished in 11 s, and then the process logged nothing for six minutes until systemd killed it. No stage after the drain logs, so we cannot tell which one held it.

Version

swamp 20261031.063736.0-sha.e459d219, swamp serve --remote-only --hot-reload --auth-mode token, filesystem datastore, Linux x86_64, run as a systemd unit with TimeoutStopSec=360. The datastore was large at the time: about 13,800 auto-definitions under @swamp/ssh and about 24,000 outputs (see #4280 and #4285).

Journal (2026-11-02, UTC)

06:55:27–06:55:41  hourly retention pass ran and finished (workflowRuns removed 184)
06:56:23  systemd: Stopping orbit-swamp-serve.service
06:56:23  Shutting down, draining in-flight runs for up to 30s...
06:56:23  Draining 1 active run(s)...
06:56:34  Dispatch "cfe13408-…" on worker "swamp-role-209-b" "failed" in 20684ms
06:56:34  Scheduled execution service stopped
06:59:42  Detached workflow run rejected: "Serve is shutting down; run … was not started"
07:00:56  Detached workflow run rejected: "Serve is shutting down; run … was not started"
07:02:23  systemd: State 'stop-sigterm' timed out. Killing.
07:02:24  systemd: Main process exited, code=killed, status=9/KILL

Serve answered /ready with shutting_down for the whole six minutes, so all new work was refused.

Where it can wait

After runShutdownDrain and interruptShutdownRuns, the shutdown awaits these without the shutdown deadline: runIndexWarmup.finished(), datastoreApiLocks.dispose(), workerGcService.dispose() and retentionGcService.dispose() (both loop while (this.#running) with no bound), the continuation sweep, the grants, access, runtime and config pollers' stop(), policySnapshotLoader.dispose(), the telemetry flush and the audit flush. The retention pass is ruled out: it ended 42 s before the stop.

The same serve on a fresh, small datastore stopped in about 15 s.

Expected

Shutdown ends within the drain plus its documented margin, whatever the datastore size. Every stage after the drain takes its bound from ShutdownDeadline, and each one logs when it starts and ends, so a slow stage can be named.