Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#1514 Cross-instance client re-attach for multi-instance swamp serve

Opened by stack72 · 8/3/2026· Shipped 8/3/2026

Problem

In a multi-instance swamp serve deployment behind a load balancer, when a client's instance dies or their connection drops, they reconnect through the LB and hit a different instance. Today, run.attach returns not_found because the run is not in the new instance's in-memory ActiveRunRegistry. The client has no way to find their run or know what happened to it.

With active run records now in the control-plane store (#1507), every instance has visibility into all active runs across the cluster. This issue uses that data to give clients a useful response and automatic reconnection.

Desired Outcome

When a client's WebSocket drops, the CLI automatically reconnects through the LB, re-attaches to their run, and resumes streaming events. If the run was on a dead instance, the client is told it was interrupted and can check the final status.

Design — Server Side

Smarter run.attach handler

When run.attach doesn't find the run in the local ActiveRunRegistry, instead of returning not_found immediately, fall back to the control-plane store:

  1. Check controlPlaneStore.get("active-runs/{runId}")
  2. If not found: return not_found (run never existed or already completed and cleaned up)
  3. If found: read the instanceId from the record, check that instance's heartbeat
  4. Heartbeat fresh (instance alive): return run_elsewhere response — the run is executing on another instance, the client should retry through the LB
  5. Heartbeat stale (instance dead): return run_interrupted response — the run was on a dead instance, continuous reconciliation will mark it as failed

New protocol response types

Add to ServerMessage in src/serve/protocol.ts:

run_elsewhere: { type: "run.elsewhere", id: requestId, payload: { runId, instanceId } }

Tells the client the run exists but is on a different live instance. Client should retry through the LB.

run_interrupted: { type: "run.interrupted", id: requestId, payload: { runId, instanceId, reason: "instance_dead" } }

Tells the client the run was on a dead instance and is being reconciled. Client should poll workflow.history.get for the final status.

Where to add the fallback

In the handleRunAttach function in src/serve/connection.ts. Today it does:

const run = ctx.activeRunRegistry?.get(payload.runId); if (!run) { sendError(socket, requestId, "not_found", ...); return; }

After this change:

const run = ctx.activeRunRegistry?.get(payload.runId); if (!run) { // Fall back to control-plane store for cross-instance lookup if (ctx.controlPlaneStore) { const record = await ctx.controlPlaneStore.get("active-runs/" + payload.runId); if (record) { const parsed = JSON.parse(new TextDecoder().decode(record)); // Check owning instance's heartbeat const heartbeat = await ctx.controlPlaneStore.get("heartbeats/" + parsed.instanceId); if (heartbeat) { const hb = InstanceHeartbeatService.parseRecord(heartbeat); if (hb && !InstanceHeartbeatService.isStale(hb)) { send(socket, { type: "run.elsewhere", id: requestId, payload: { runId: payload.runId, instanceId: parsed.instanceId } }); return; } } // Instance dead or heartbeat missing send(socket, { type: "run.interrupted", id: requestId, payload: { runId: payload.runId, instanceId: parsed.instanceId, reason: "instance_dead" } }); return; } } sendError(socket, requestId, "not_found", ...); return; }

The handleRunAttach function needs to become async (or stay async if it already is) since the control-plane store calls are async.

controlPlaneStore on ConnectionContext

Verify that controlPlaneStore is accessible from the connection handler. It may need to be added to ConnectionContext if not already there. Check src/serve/handlers/shared.ts.

Design — Client Side

Automatic reconnection in remote_run.ts

The CLI client in src/cli/remote_run.ts needs reconnection logic. Today, a WebSocket drop causes the async iterator to end or throw. After this change:

  1. On WebSocket drop during workflow.run or run.attach:

    • Remember the runId (from the started event) and last seen seq number
    • Reconnect to the same server URL (routes through LB to any instance)
    • Send run.attach with { runId, afterSeq }

  2. Handle new response types:

    • run.attached: resume consuming events (existing behavior)
    • run.elsewhere: wait 1-2 seconds, retry run.attach (same reconnect URL, LB routes to different instance)
    • run.interrupted: log that the run was interrupted, poll workflow.history.get for final status, report to user
    • not_found: after N retries, give up and report the run could not be found
    • error: standard error handling

  3. Retry limits:

    • Max retries for run.elsewhere: ~10 (with 2-3 instances, should hit the right one within a few tries)
    • Max retries for reconnection: ~5 (if the server is completely down, give up)
    • Backoff between retries: 1-2 seconds

  4. User-visible behavior:

    • On reconnection: "[reconnecting...]" status line
    • On run.elsewhere: "[run is on another instance, retrying...]"
    • On run.interrupted: "Run was interrupted (instance died). Checking final status..."
    • On success: events resume seamlessly from where they left off (via afterSeq)

What the client needs to remember

The client already receives the runId from the started event and seq numbers on every event frame (shipped in Phase 1). It just needs to store them instead of discarding them:

let currentRunId: string | undefined; let lastSeq = 0;

// In the event handler: if (event.kind === "started") currentRunId = event.runId; if (event.seq) lastSeq = event.seq;

runWorkflowOverServer changes

The main function in remote_run.ts that the CLI uses. Needs a reconnection wrapper around the WebSocket lifecycle:

async function* runWorkflowOverServer(opts) { let runId: string | undefined; let lastSeq = 0; let retries = 0;

while (retries < MAX_RETRIES) {
  try {
    const ws = connect(opts.server);
    if (!runId) {
      // First connection — start the workflow
      send(ws, { type: "workflow.run", ... });
    } else {
      // Reconnection — re-attach
      send(ws, { type: "run.attach", payload: { runId, afterSeq: lastSeq } });
    }
    for await (const msg of receive(ws)) {
      if (msg.type === "event") {
        if (msg.event.kind === "started") runId = msg.event.runId;
        if (msg.event.seq) lastSeq = msg.event.seq;
        yield deserializeEvent(msg.event);
      }
      if (msg.type === "run.elsewhere") {
        // Break inner loop, retry in outer loop
        await delay(1500);
        retries++;
        break;
      }
      if (msg.type === "run.interrupted") {
        // Run is dead, poll for final status
        yield { kind: "interrupted", reason: msg.payload.reason };
        return;
      }
      if (msg.type === "done") return;
      if (msg.type === "error") throw new Error(msg.error.message);
    }
  } catch (err) {
    if (!runId) throw err; // Can't reconnect without a run ID
    retries++;
    await delay(1000 * retries);
  }
}

}

Testing

Server tests

  1. run.attach with run in local registry: existing behavior unchanged
  2. run.attach miss, active-runs record exists, owning instance alive: returns run_elsewhere
  3. run.attach miss, active-runs record exists, owning instance dead: returns run_interrupted
  4. run.attach miss, no active-runs record: returns not_found
  5. run.attach miss, active-runs record exists but no heartbeat for owner: returns run_interrupted
  6. run.attach miss, control-plane store not available: returns not_found (graceful degradation)

Client tests

  1. WebSocket drop during workflow.run: reconnects and sends run.attach
  2. run.elsewhere response: retries after delay
  3. run.interrupted response: reports interrupted status
  4. Multiple retries until hitting the right instance
  5. Max retries exceeded: gives up with error
  6. No runId available (drop before started event): does not attempt reconnect

Integration tests

  1. Binary test: start workflow on instance, kill instance, verify client reconnects and gets interrupted status

Files to modify

Server

  • src/serve/connection.ts — update handleRunAttach with control-plane store fallback
  • src/serve/protocol.ts — add run.elsewhere and run.interrupted message types

Client

  • src/cli/remote_run.ts — reconnection logic, retry loop, new response handling

Parent issue: #1448

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 5 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/3/2026, 4:44:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/3/2026, 1:14:48 PM

Sign in to post a ripple.