Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#1491 Startup cache hydration for swamp serve instance replacement

Opened by stack72 · 8/1/2026· Shipped 8/1/2026

Problem

When swamp serve runs as a disposable instance (AWS ASG, Kubernetes pod) with a remote datastore (S3), the instance can die and be replaced. Phase 3 (#2036) added heartbeat-based cross-machine reconciliation — the replacement instance detects the dead predecessor via stale heartbeats in the control-plane store and marks its interrupted runs as failed.

However, the reconciliation reads WorkflowRun YAML files from the local cache to find runs tagged with the dead instance's instanceId. On a fresh replacement machine, the local cache is empty — the YAML files are in S3 but haven't been pulled into the local cache yet. The reconciliation finds nothing, and the dead instance's runs stay stuck in status: running forever.

Today, the operator must add swamp datastore sync --pull to their cloud-init, Kubernetes init container, or systemd ExecStartPre before starting swamp serve. This works but it's an operational burden that shouldn't exist — the operator shouldn't need to know about cache hydration to get HA working.

Desired Outcome

When swamp serve starts with --detach-runs and a remote datastore, it automatically hydrates the local cache from the remote datastore before running boot reconciliation. The operator just starts serve — no init script, no manual sync, no magic incantation.

Design

What needs to be pulled

The reconciliation only needs workflow-runs/ directory — the WorkflowRun YAML files that contain status and instanceId. It does NOT need model data, outputs, audit logs, telemetry, or evaluated definitions at boot time. Those are pulled lazily during execution via per-model lock acquisition (existing behavior).

How the sync service works today

swamp serve calls requireInitializedRepoUnlocked which creates the sync service but does NOT call pullChanged. Each workflow execution acquires per-model locks which trigger scoped pulls. There is no startup pull.

The sync service has pullChanged(options?: DatastoreSyncOptions) which pulls everything from the remote. It also supports scoped pulls via SyncContext with a models array — but this scopes by model, not by directory. There's no directory-level scoping today.

Implementation approach

Add a targeted pull at serve startup, BEFORE boot reconciliation runs. The pull should happen only when:

  1. --detach-runs is active
  2. A remote sync service exists (not filesystem datastore)

Two options for scoping:

Option A — Full pull. Call pullChanged() with no scope. Downloads everything from S3 into the local cache. Simple but potentially slow on large datastores. The zero-diff fast path makes subsequent boots instant (only the first boot on a fresh machine pays the full cost). This is the simplest implementation.

Option B — Directory-scoped pull. Only pull workflow-runs/. Faster on large datastores but requires adding directory-level scoping to pullChanged or implementing a separate targeted pull mechanism. More complex.

Recommendation: Start with Option A (full pull). If performance is a problem for large datastores, optimize later with directory-level scoping. The zero-diff fast path means this cost is only paid once per fresh machine.

Where in the startup sequence

The pull must happen AFTER the sync service is created but BEFORE:

  • reapOrphanedWorkflowRuns (reads WorkflowRun YAML from local cache)
  • reconcileRemoteInterruptedRuns (reads WorkflowRun YAML from local cache)
  • replayPendingRuns (reads pending runs from control-plane store — this doesn't need the pull, but should run after reconciliation)

Current startup order in serve.ts:

  1. Create sync service (requireInitializedRepoUnlocked)
  2. Create run tracker
  3. Reap stale tracker runs
  4. Reap dead process runs (--detach-runs)
  5. Reap orphaned workflow runs ← needs YAML in cache
  6. Reconcile remote interrupted runs ← needs YAML in cache
  7. Sweep stale records (leases, dispatches, workers)
  8. Start Deno.serve
  9. Replay pending runs
  10. Start heartbeat

The pull should go between steps 1 and 3:

  1. Create sync service
  2. NEW: Pull from remote datastore if --detach-runs + remote sync service
  3. Reap stale tracker runs ... (rest unchanged)

Code changes

In src/cli/commands/serve.ts, after the sync service is created and before the run tracker reaping:

if (detachRuns && syncService) {
  logger.info("Hydrating local cache from remote datastore...");
  try {
    const pulled = await syncService.pullChanged();
    if (typeof pulled === "number" && pulled > 0) {
      logger.info`Pulled ${pulled} file(s) from remote datastore`;
    }
    // Invalidate the catalog so it rebuilds from the fresh cache
    repoContext.catalogStore.invalidate();
  } catch (err) {
    logger.warn("Startup cache hydration failed: {error}", {
      error: err instanceof Error ? err.message : String(err),
    });
    // Continue anyway — reconciliation will be incomplete but serve
    // should still start. The operator can run manual sync later.
  }
}

Key details:

  • Call catalogStore.invalidate() after the pull so the catalog rebuilds from the fresh data
  • Catch and warn on failure — serve should start even if the pull fails (degraded but operational)
  • Log the result so the operator sees what happened
  • The pullChanged call respects the datastore's namespace configuration automatically

Testing

  1. Unit test: verify pullChanged is called when detachRuns + syncService exist
  2. Unit test: verify pullChanged is NOT called without --detach-runs
  3. Unit test: verify pullChanged failure is caught and logged (serve continues)
  4. Unit test: verify catalogStore.invalidate() is called after successful pull
  5. Binary verification: start serve with --detach-runs and S3 datastore on a fresh machine (empty cache), verify WorkflowRun YAML files are present in the local cache after startup

What this unlocks

With this change, the full instance replacement flow works without operator intervention:

  1. Instance A runs serve with --detach-runs and S3 datastore
  2. Instance A runs workflows, writes heartbeat to S3
  3. Instance A dies (spot termination, OOM, etc.)
  4. Instance B starts serve with --detach-runs and S3 datastore
  5. NEW: Serve pulls workflow-runs/ from S3 into local cache
  6. Serve detects A's stale heartbeat in S3
  7. Serve finds A's WorkflowRun YAML files in local cache (now available)
  8. Serve calls interrupt("server_crash") on A's running workflows
  9. Serve replays pending webhook/cron runs from S3 control-plane store
  10. Serve starts accepting traffic

No init script. No manual sync. Just start serve.

Files to modify

  • src/cli/commands/serve.ts — add startup pull between sync service creation and reconciliation
  • Possibly src/cli/commands/serve.ts test coverage for the startup sequence

Scope

This is ONLY the startup pull. It does not change:

  • The sync service interface
  • The pull scoping mechanism
  • The existing per-execution lazy pull behavior
  • The control-plane store (heartbeats, pending runs are read directly from S3, not from the cache)

Parent issue: #1448

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 3 MOREFINDINGS+ 4 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/1/2026, 11:23:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/1/2026, 10:19:44 PM

Sign in to post a ripple.