#3961 swamp serve on a v3 datastore: a run interrupted at a clean shutdown is never committed when the bucket is slow, and stays running for every other checkout

Summary

Found while triaging #3958 (the format 2 version of this bug). On a format 3 (v3) datastore replicated to S3, swamp serve stopped cleanly with SIGTERM while a workflow run is mid-step does not save the run's interrupted record anywhere when the bucket is slow, not even in the local commits.db. Every other checkout keeps reading the run as running, and the shutdown takes about 91 s, past systemd's default TimeoutStopSec of 90 s.

Environment

• swamp 20261026.014016.0-sha.7e1aad04 (Linux x86_64), @swamp/s3-datastore 2026.10.24.2, swamp datastore setup extension @swamp/s3-datastore --format 3, Litestream 0.5.17.
• The local fake S3 from #3958 (fake_s3.py), with batch delete (POST ?delete) added because Litestream's cleanup needs it. Every PutObject is held 15 s at shutdown. No AWS.

Reproduction

The #3958 scripts with --format 3 at setup and a 15 s hold:

1. Start serve and run workflow slow (first, then sleep 120, then after) through it. Wait until step sleep is running.
2. Hold every PutObject 15 s. Send SIGTERM to serve and wait for it to exit.
3. Read the run from a second checkout: a copy of the repo without .swamp/, with a fresh SWAMP_HOME. It restores from the replica.

Expected

The interrupted record is committed locally before anything waits on the bucket, and the shutdown finishes within its drain budget. Another checkout reads interrupted.

Actual

Serve 1 (SIGTERM 13:46:32, exit 0 at 13:48:03; the no-delay control exits in 30 s):

Shutting down, draining in-flight runs for up to 30s...
Draining 1 active run(s)...
Aborting 1 undrained run(s)...
Interrupted workflow run "53bf95b3-…" (server shutdown)
A workflow run checkpoint failed; the run goes on and its end commit tries again: S3OperationError [AbortError]: S3 putObject failed key=_blobs/_default/sha256/71/712cba20… aborted AbortError — Request aborted
   (cause at Object.close (v3_datastore.ts:1173) ← closeV3Datastores (v3_repo_context.ts:1540))
The final record of 1 workflow run(s) was not committed; run swamp workflow recover once this process has exited to settle them: "53bf95b3-…"
Error: This command's changes are saved locally, but swamp could not confirm that they reached the datastore's replica: the sync did not finish within 10s. ...

A second checkout reads running [first: succeeded, sleep: running, after: pending]. The no-delay control reads interrupted.

Analysis

• A v3 commit uploads its content chunks (_blobs/...) before it commits locally. The interrupt's chunk uploads ran one after another, each held 15 s. close() in src/infrastructure/datastore_v3/v3_datastore.ts calls lifetime.abort() and cancels the upload still in flight, so the record is never committed. A slow bucket at shutdown therefore loses the interrupt outright.
• The upload of the record is not bounded by the drain budget, so shutdown ran 61 s past the 30 s drain.
• After a restart of the same checkout, the record is gone, so whether the run settles depends on the run tracker. Within 90 s of the last tracker heartbeat it does not (see the companion issue on the v3 reaper).

Suggested direction

Commit the shutdown interrupt locally without waiting on the bucket, or bound its uploads by what is left of the drain budget and keep a local marker on a miss, so the next start commits it.