#1627 serve: cron ticks leak pending-run records, replaying every past tick on restart

Summary

Every cron tick in swamp serve writes a durable intent record to the control-plane store at .swamp/_control/pending-runs/ and never retires it, even when the run it spawned completes successfully. The directory grows by one record per tick for as long as serve is up, and the next boot replays the entire accumulated history back-to-back.

Two code paths touch that directory and only one of them cleans up:

• Boot replay — reads every pending record, executes it, deletes it.
• Live tick — writes a record, executes the run, and never deletes it.

The dual-write has a single-sided delete. SQLite's pending_runs table is retired correctly on completion; the control-plane copy on the filesystem never is. Boot replay merges both sources, so the filesystem side replays every tick since the store was created.

Version

Reproduced on 20260812.183910.0-sha.57b1d29b (also present on 20260809.004828.0-sha.b61c9de2). macOS 25.5.0, local-filesystem datastore, single swamp serve instance.

Reproduction

A workflow with trigger.schedule set to * * * * *, serve started fresh with an empty pending-runs/, polled every 20s:

19:19:58  pending=0  sqlite=0   <- baseline, clean queue
19:20:23  pending=1  sqlite=0   <- tick 1, run succeeded
19:21:03  pending=2  sqlite=0   <- tick 2
19:22:03  pending=3  sqlite=0   <- tick 3
19:23:04  pending=4  sqlite=0   <- tick 4
19:24:04  pending=5  sqlite=0   <- tick 5

Where the two counts come from:

ls .swamp/_control/pending-runs/ | wc -l
sqlite3 .swamp/run_tracker.db "select count(*) from pending_runs"

Five ticks, five records, zero retired, while serve was still running and healthy. swamp run history shows all five workflow runs and their ten child method runs as completed — this is not a crash-recovery artifact, successful runs leak identically.

The leaked records carry no status, no completion marker, and no runId linking them to the run they spawned:

{"id":"268da566-837e-4ba2-b0c6-16c1d50e77f4","source":"cron","workflowIdOrName":"slack-triage","createdAt":"2026-08-12T19:20:00.023Z"}

It's a write-ahead intent with no way to tell whether it was already carried out.

Replay confirmed

Restarting serve with two leaked records present:

[INF] serve: Boot: reconciling workflow run state
[INF] serve·boot-reconciliation: Replayed pending cron run for "slack-triage"
[INF] serve·boot-reconciliation: Replayed pending cron run for "slack-triage"
[INF] serve: Replayed 2 pending run(s) from previous process

Both had already run to completion under the previous process. Boot deletes them after replaying, which is why the backlog drains on restart — and why the symptom presents as a burst of duplicate runs rather than as a growing directory.

No backstop in local-filesystem mode

[INF] serve: HA: detached runs, pending-run durability (no remote control-plane — heartbeat and reconciliation disabled)

The durable pending-run write is active, but the continuous reconciliation that would sweep leaked control-plane records is disabled. Single-instance local deployments have no cleanup path at all. This may be why the bug isn't visible in the multi-instance S3 deployments the feature was designed for — there, reconciliation is running.

Impact

On a */5 8-17 * * * schedule this accrues ~120 records/day. A Mon→Wed uptime left ~450 records, all of which replayed on the next restart at roughly 1/sec — a burst of several hundred duplicate workflow runs, each re-executing real side effects (in this repo: Slack reads, LLM calls, and push notifications). Earlier restarts of the same process drained backlogs of 136 and 73.

swamp run doctor does not cover this

run doctor reported "No active or stale runs" with 136 pending records sitting on disk waiting to replay, and again with 5 leaked records during the repro above. There is no warning before the next replay storm.

Related

• #1505 introduced the mechanism — "Durable pending runs dual-written to SQLite + control-plane store", "Pending run replay merged from SQLite + remote". This bug falls out of that dual-write.
• #1507 designs delete-on-complete, but only for active-runs/, not pending-runs/. It flags cron as the awkward path: "For paths 4-5 … delete … in the completion handler (for webhook/cron paths)."
• #1493 is the continuous reconciliation backstop — disabled in local-filesystem mode, per the log line above.

Also observed

.swamp/_control/active-runs/ accumulates empty directories that are never removed — 6 stale ones on this repo, the oldest two days old, with run doctor reporting clean throughout. Less severe than the pending-run leak (they don't replay), but the same missing-cleanup shape.

Workaround

Delete the contents of .swamp/_control/pending-runs/ before restarting serve.