Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#1507 Track active runs in the control-plane store for cross-instance visibility

Opened by stack72 · 8/2/2026· Shipped 8/2/2026

Problem

In a multi-instance swamp serve deployment, each instance only knows about runs in its own in-memory ActiveRunRegistry. When a client reconnects through a load balancer and hits a different instance, that instance has no way to know the run exists — it returns not_found even though the run is actively executing on another instance.

This also means operators have no cross-cluster view of active runs. Running swamp run history --active on instance B only shows B's runs, not A's.

Desired Outcome

Every active run is recorded in the control-plane store (S3) so any instance can discover it. This is the foundation for cross-instance client re-attach (Phase C) and cross-cluster observability.

Design

Write on run start

When a workflow run or model method run starts on any instance, write a record to the control-plane store:

Key: active-runs/{runId} Value: { instanceId, workflowName, startedAt }

This happens in the detached task that spawns the execution — after the started event fires and the run ID is known. The write is best-effort (fire-and-forget with logged .catch) — it should not block or fail the execution.

Delete on run complete

When a run reaches a terminal state (succeeded, failed, cancelled), delete the record:

controlPlaneStore.delete("active-runs/{runId}")

Also best-effort with logged .catch. If the delete fails, reconciliation cleanup handles it.

Reconciliation cleanup

When continuous reconciliation (#1493) finds a stale instance and claims it, also clean up that instance's active-runs/ entries:

  1. List active-runs/ from the control-plane store
  2. For each entry, check if the instanceId matches a stale (claimed) instance
  3. Delete matching entries

This prevents active-runs/ records from accumulating for dead instances.

Where to write the records

There are four execution paths that create runs in serve:

  1. handleWorkflowRun (detached path) in src/serve/handlers/workflow_handlers.ts — workflow run via WebSocket
  2. handleWorkflowResume (detached path) in src/serve/handlers/workflow_handlers.ts — workflow resume via WebSocket
  3. handleModelMethodRun (detached path) in src/serve/handlers/model_handlers.ts — model method run via WebSocket
  4. WebhookService.executeWorkflow in src/serve/webhook.ts — webhook-triggered workflow run
  5. ScheduledExecutionService.executeWorkflow in src/libswamp/workflows/scheduled_execution.ts — cron-triggered workflow run

For paths 1-3, the detached task already writes to the ActiveRunRegistry. The control-plane store write goes alongside the register call.

For paths 4-5, execution goes through executeWorkflowWithLocks which eventually calls the execution service. The control-plane store write could go in the onEvent callback when the started event arrives with the runId.

The simplest approach: write the active-runs record in the same place that the ActiveRunRegistry.register call happens (for detached paths) or when the started event fires (for webhook/cron paths). Delete when the run finishes — in the detached task's finally block (for detached paths) or in the completion handler (for webhook/cron paths).

ConnectionContext change

Add controlPlaneStore to the data available in handlers. It's already on ConnectionContext — verify the handlers that need it can access it.

What this does NOT include

  • Smarter run.attach response — that's a separate piece that consumes these records
  • Client reconnection logic — separate piece
  • Changes to run.attach handler — separate piece

This is purely "write the data." Consumers come later.

Record shape

{ "instanceId": "uuid-of-owning-instance", "workflowName": "deploy-pipeline", "runKind": "workflow-run", "startedAt": "2026-08-01T12:00:00Z" }

Keep it small — this is read frequently by run.attach (future) and should be a fast S3 GET.

Testing

  1. Unit test: active-runs record written when detached workflow run starts
  2. Unit test: active-runs record deleted when detached workflow run completes
  3. Unit test: active-runs record deleted when detached workflow run fails
  4. Unit test: active-runs record written for webhook-triggered run
  5. Unit test: active-runs record written for cron-triggered run
  6. Unit test: reconciliation cleanup deletes active-runs for stale instances
  7. Unit test: write failure does not prevent run from executing
  8. Unit test: delete failure does not prevent run from completing

Files to modify

  • src/serve/handlers/workflow_handlers.ts — write/delete in detached task
  • src/serve/handlers/model_handlers.ts — write/delete in detached task
  • src/serve/webhook.ts — write/delete around executeWorkflow
  • src/libswamp/workflows/scheduled_execution.ts — write/delete via callback or hook
  • src/serve/boot_reconciliation.ts — add active-runs cleanup to reconciliation

Parent issue: #1448

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 5 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/2/2026, 11:50:22 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/2/2026, 10:29:31 PM

Sign in to post a ripple.