Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4576 swamp serve HA: a pending-run sweep can replay a queued run twice across hosts (heartbeat read error counts as a dead owner; control-plane-only runs have no claim compare-and-swap)

Opened by stack72 · 11/10/2026· Shipped 11/10/2026

Follow-up to swamp-club#4566 (PR swamp-club/swamp#3706), which gives queued webhook and cron runs an owner and adds a sweep (PendingReplaySweep, src/serve/pending_run_owner.ts) that replays a gone owner's runs. Two gaps remain when several hosts share one remote control plane (HA). Both are narrower than the behaviour before #4566, where every HA instance restart replayed all of its live peers' queued runs.

1. A heartbeat that cannot be read counts as a dead owner

pendingOwnerLiveness returns false when the heartbeat get throws or times out (5 s withDeadline). The sweep's liveness was configured to be cautious (unjudgedOwner: "alive" when there is no heartbeat store), but a read error falls through to "gone". On a bucket blip during the sweep's control-plane pass (about once a minute), a live peer's queued run can be claimed and replayed while the peer also runs it from its in-memory queue.

Fix: in the sweep, treat a heartbeat read error or timeout as alive (catch { return deps.unjudgedOwner === "alive"; }); keep a missing or stale heartbeat as gone. Boot keeps its current behaviour.

2. Two live sweepers can both claim a control-plane-only run

claimPendingRun (src/serve/boot_reconciliation.ts) takes a run held only in the control plane with a plain put; only runs in the local run tracker get a compare-and-swap. When an HA peer crashes with queued runs, every surviving instance's sweep may judge it gone in the same window, and two can each replay the same run.

Fix: claim through putIfAbsent on a claim key such as pending-claims/<id>/<previous owner instance id> where the store supports it (as fire-records/ does for cron fires), or re-read the entry after the put and replay only if this instance's owner stuck. Add a model-based property test of concurrent sweepers over a shared in-memory control plane.

Acceptance

  • Two sweepers racing on one dead peer's control-plane-only run replay it exactly once.
  • A heartbeat read error during a sweep never replays a live peer's run.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 2 MOREISSUE_LINKED+ 6 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 8:13:51 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/10/2026, 7:24:50 AM
stack72 linked sibling of #457711/10/2026, 7:25:10 AM
stack72 removed sibling of #457711/10/2026, 7:27:58 AM

Sign in to post a ripple.