RUN A MULTI-INSTANCE DEPLOYMENT

This guide shows you how to run swamp serve in a multi-instance deployment where two or more instances share a datastore behind a load balancer.

Prerequisites

• A shared datastore (e.g. S3) accessible by all instances
• A load balancer or reverse proxy distributing client connections
• TLS certificates and an authentication mode (token or oauth) — required for off-loopback binding

Enable detached runs

Start each instance with --detach-runs so runs survive client disconnection and are visible to peers for reconciliation:

swamp serve \
  --host 0.0.0.0 \
  --auth-mode token \
  --cert-file /etc/swamp/tls/cert.pem \
  --key-file /etc/swamp/tls/key.pem \
  --detach-runs

Without --detach-runs, runs are tied to the client connection and die with the process. The reconciliation flags have no effect.

Tune the timing constants

Three flags control how quickly a dead instance is detected and its runs recovered. All three only take effect when --detach-runs is active.

swamp serve \
  --detach-runs \
  --heartbeat-interval 30s \
  --stale-ttl 90s \
  --reconciliation-interval 60s \
  # ... other flags

Adjust these based on your deployment's characteristics:

Low-latency datastore (local SSD, same-region S3): The defaults work well. A dead instance is detected within ~90s and its runs reconciled within ~150s.

High-latency or unreliable datastore (cross-region S3, high-jitter networks): Increase --heartbeat-interval to allow for write latency, and increase --stale-ttl proportionally (must remain at least 2× --heartbeat-interval) to avoid false-positive death detection:

--heartbeat-interval 60s \
--stale-ttl 180s \
--reconciliation-interval 120s

Fast recovery priority: Decrease --reconciliation-interval so surviving instances scan for orphaned runs more often. The heartbeat interval and stale TTL control detection speed; the reconciliation interval controls adoption speed:

--heartbeat-interval 15s \
--stale-ttl 45s \
--reconciliation-interval 30s

The tradeoff is more frequent writes and scans against the shared store.

Verify reconciliation is working

If an instance is stopped or killed, its peers should detect the stale heartbeat and adopt its orphaned runs. Check with swamp run history --server against a surviving instance to confirm the runs were picked up.

Related

• Serve Flags — full flag and environment variable reference for --detach-runs, --heartbeat-interval, --stale-ttl, and --reconciliation-interval
• swamp serve — design decisions behind heartbeats, reconciliation, and crash safety
• The Run Tracker — per-instance liveness tracking