#3535 Retention under serve load: measure and fix automatic passes against #3518's busy-serve contention, and the large-backlog first pass

Background

swamp-club#3473 adds .swamp.yaml garbageCollection retention with an optional automatic: true:

• serve runs a pass five minutes after start, then hourly, in batches of 100, taking the sync gate per batch, with a GcMutex shared with run.gc and data.gc;
• the CLI runs a pass at the end of a command, at most hourly.

swamp-club#3518 is the incident this has to avoid. On serve 20261015.173228.0-sha.95cd2318 with a local filesystem datastore and about 36 remote workers busy, a scheduled swamp run gc + swamp data gc through serve took 26 minutes. Throughout, worker dispatch results timed out, scheduled workflows queued, and webhooks were rejected with Queue full.

Measured in review: the same backlog shape (220 workflow runs with 5 steps, 1,100 outputs, 220 snapshots, about 11.9 MB) on a standalone repo takes 2.4 s for run gc and 1.8 s for data gc with that released binary, and 2.2 s / 1.8 s with #3473's branch. So #3518's time isn't the collection work. It's collection through serve contending with workers' result reports, schedules and webhooks.

#3473's per-batch gate is meant to prevent that, but it hasn't been measured under load. Until it is, automatic: true shouldn't be turned on for a busy production serve.

What to do

1. Reproduce #3518 locally: serve on a filesystem datastore with about 30 workers each running a long step (the swamp-uat fleet recipe), a backlog of about 220 runs with outputs, and a run gc + data gc through serve (--server).
   • Measure: the pass's wall time; worker dispatch-result latency and timeouts during it; scheduled-run start delay; webhook Queue full rejections.
   • Compare the released binary with current main.
2. Then the same load with automatic: true and the same backlog: serve's own hourly pass instead of the manual commands.
3. Fix what it shows, for example lock hold time, gate granularity or batch size. Say which, and keep run gc and data gc correct.
4. Large backlog: about 100k runs and about 140k outputs.
   • Measure: the first automatic pass's time, how many hourly passes catch-up takes, and serve write latency during a pass.
   • Runs and outputs rescan from the start each batch, which is quadratic. If that dominates, resume each kind's scan from the last key, as data's created_at cursor does.

Acceptance

• With automatic: true on a serve running about 30 busy workers and #3518's backlog, no dispatch result times out, scheduled runs start within their normal delay, and no webhook is rejected while a pass runs.
• The large-backlog figures are in the PR, and design/surfaces/repo.md says how long a first catch-up takes.

Blocked by: swamp-club#3473 merging. Related: swamp-club#3518 (this should close it, or say what's left).