Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#1493 Continuous reconciliation for multi-instance swamp serve

Opened by stack72 · 8/1/2026· Shipped 8/2/2026

Problem

Stale heartbeat detection runs once at boot (reconcileRemoteInterruptedRuns in src/serve/boot_reconciliation.ts). This works for single-instance — the replacement boots, finds the dead predecessor, reconciles. But with N instances running behind a load balancer, instance A can die at any time while instances B and C are serving traffic. B and C don't notice until they restart — which might be never. A's runs sit in status: running forever, A's stale heartbeat sits in S3 forever, and no one cleans up.

Desired Outcome

Surviving instances detect a peer's death while running, not just when they boot. A's orphaned runs are reconciled within ~60 seconds of its heartbeat going stale, regardless of whether B or C restart.

Design

Periodic reconciliation timer

A recurring timer on each instance calls the existing reconcileRemoteInterruptedRuns function every ~60 seconds. This is the same function that runs at boot — it lists heartbeats from the control-plane store, checks for stale ones (heartbeatAt > 90 seconds old), and interrupts matching runs.

The timer only runs when --detach-runs is active and a control-plane store is available. Uses Deno.unrefTimer so it doesn't prevent shutdown.

Jitter to reduce races

Without jitter, all instances fire their timers at roughly the same time and detect the same stale heartbeat simultaneously. Add 0-500ms of crypto-random jitter per tick to stagger the checks:

const jitter = new Uint32Array(1);
crypto.getRandomValues(jitter);
const jitterMs = (jitter[0] / 0xFFFFFFFF) * 500;
setTimeout(reconcile, 60_000 + jitterMs);

Each tick schedules the next with fresh jitter (not setInterval with a fixed offset). The jitter is a tiebreaker — putIfAbsent is the real safety net.

Do NOT use Math.random() — it's a PRNG seeded from process startup and two instances starting from the same image at the same time could get similar sequences. Use crypto.getRandomValues for truly independent randomness across instances.

Per-instance claims

When reconciliation finds a stale instance, before interrupting its runs, claim it with putIfAbsent at claims/reconcile-instance/{deadInstanceId}. If putIfAbsent returns true, proceed with reconciliation. If false, another instance already claimed it — skip.

This prevents two instances both reconciling the same dead peer's runs simultaneously. The jitter makes races rare; the claim makes them safe.

The claim record value should include the claiming instance's ID and timestamp:

{ claimedBy: instanceId, claimedAt: new Date().toISOString() }

Claim TTL and cleanup

Claims are ephemeral. If the claiming instance dies mid-reconciliation, the claim must expire so another instance can retry. TTL: 5 minutes.

A periodic cleanup (piggyback on the reconciliation timer, or a separate less-frequent timer) lists claims/reconcile-instance/ and deletes entries older than the TTL. This runs on every tick alongside the heartbeat check.

Single-node behavior

In single-node mode, the timer fires, lists heartbeats, finds only its own (fresh, skipped via the instanceId === self guard), and returns 0. No claims created, no runs touched, no S3 writes. One list call per minute — trivial cost. The continuous reconciliation is a harmless no-op in single-node mode.

Changes to reconcileRemoteInterruptedRuns

The existing function in src/serve/boot_reconciliation.ts needs one addition: per-instance claiming before interrupting runs. The current flow is:

  1. List heartbeat keys
  2. Parse each, skip self, check isStale
  3. Collect stale instance IDs
  4. Find WorkflowRun YAML files with matching instanceId and status: running
  5. Call interrupt("server_crash") on each
  6. Delete stale heartbeat keys

The new flow adds a claim step between 3 and 4:

  1. List heartbeat keys
  2. Parse each, skip self, check isStale
  3. Collect stale instance IDs
  4. NEW: For each stale instance, putIfAbsent at claims/reconcile-instance/{id}. If false, skip this instance.
  5. Find WorkflowRun YAML files with matching instanceId and status: running
  6. Call interrupt("server_crash") on each
  7. Delete stale heartbeat keys

The function signature may need an additional parameter for the claim prefix, or it can be hardcoded since the key format is internal.

The boot path also benefits from claims (defense in depth for ASG replacement overlap), so the claim logic should be in the function itself, not just in the timer wrapper.

Timer implementation in serve.ts

Wire the timer after the heartbeat service starts (around line 2520 in src/cli/commands/serve.ts):

if (controlPlaneStore && instanceId) {
  const scheduleReconciliation = () => {
    const jitter = new Uint32Array(1);
    crypto.getRandomValues(jitter);
    const jitterMs = (jitter[0] / 0xFFFFFFFF) * 500;
    const timer = setTimeout(async () => {
      try {
        await reconcileRemoteInterruptedRuns({
          controlPlaneStore,
          instanceId,
          workflowRunRepo: repoContext.workflowRunRepo,
        });
        // Clean up expired claims
        // ... claim cleanup logic
      } catch (err) {
        logger.warn("Continuous reconciliation failed: {error}", {
          error: err instanceof Error ? err.message : String(err),
        });
      }
      scheduleReconciliation(); // schedule next tick
    }, 60_000 + jitterMs);
    Deno.unrefTimer(timer);
  };
  scheduleReconciliation();
}

Claim cleanup

On each reconciliation tick, after checking heartbeats, list claims/reconcile-instance/ and delete entries older than 5 minutes. This is lightweight — typically 0-2 entries.

const claimKeys = await controlPlaneStore.list("claims/reconcile-instance/");
for (const key of claimKeys) {
  const data = await controlPlaneStore.get(key);
  if (!data) continue;
  try {
    const claim = JSON.parse(new TextDecoder().decode(data));
    const age = Date.now() - new Date(claim.claimedAt).getTime();
    if (age > CLAIM_TTL_MS) {
      await controlPlaneStore.delete(key);
    }
  } catch {
    await controlPlaneStore.delete(key); // corrupt claim, clean up
  }
}

Testing

  1. Unit test: reconciliation with claims — stale heartbeat found, claim succeeds, runs interrupted
  2. Unit test: reconciliation with claims — stale heartbeat found, claim fails (already claimed), runs NOT interrupted
  3. Unit test: claim cleanup — expired claims deleted, fresh claims kept
  4. Unit test: corrupt claim records cleaned up
  5. Unit test: single-instance behavior — own heartbeat skipped, no claims created, returns 0
  6. Unit test: timer scheduling — verify jitter uses crypto.getRandomValues, not Math.random
  7. Integration: verify continuous reconciliation detects a stale heartbeat added to the control-plane store while serve is running (not just at boot)

Files to modify

  • src/serve/boot_reconciliation.ts — add claim logic to reconcileRemoteInterruptedRuns, add claim cleanup function
  • src/serve/boot_reconciliation_test.ts — new tests for claim behavior
  • src/cli/commands/serve.ts — add the reconciliation timer after heartbeat start

What this does NOT change

  • Boot reconciliation — still runs at startup, now also uses claims for safety
  • Heartbeat service — unchanged
  • The reconciliation logic itself — same interrupt() calls, same YAML matching
  • Single-instance behavior — timer runs but is a no-op

Dependencies

  • Control-plane store with putIfAbsent (shipped in #2033)
  • Instance heartbeat (shipped in #2036)
  • reconcileRemoteInterruptedRuns function (shipped in #2036)
  • Startup cache hydration (#1491) — needed so the reconciliation has YAML files to read on a fresh machine

Parent issue: #1448

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 5 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/2/2026, 9:38:25 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/1/2026, 11:38:03 PM

Sign in to post a ripple.