RUN A MULTI-INSTANCE DEPLOYMENT

This guide shows you how to run swamp serve in a multi-instance deployment where two or more instances share a datastore behind a load balancer.

Prerequisites

A shared datastore (e.g. S3) accessible by all instances
A load balancer or reverse proxy distributing client connections
TLS certificates and an authentication mode (token or oauth) — required for off-loopback binding

HA auto-detection

HA mode is detected automatically based on the datastore configuration. When the datastore supports a control plane (e.g. S3), runs are automatically detached from the client connection and reconciliation is enabled — no flag required:

swamp serve \
  --host 0.0.0.0 \
  --auth-mode token \
  --cert-file /etc/swamp/tls/cert.pem \
  --key-file /etc/swamp/tls/key.pem

Note

The --detach-runs flag is deprecated. It is still accepted for backwards compatibility but has no effect — HA mode is determined entirely by the datastore.

Tune the timing constants

Three flags control how quickly a dead instance is detected and its runs recovered. All three only take effect with a control-plane-capable datastore.

swamp serve \
  --heartbeat-interval 30s \
  --stale-ttl 90s \
  --reconciliation-interval 60s \
  # ... other flags

Adjust these based on your deployment's characteristics:

Low-latency datastore (local SSD, same-region S3): The defaults work well. A dead instance is detected within ~90s and its runs reconciled within ~150s.

High-latency or unreliable datastore (cross-region S3, high-jitter networks): Increase --heartbeat-interval to allow for write latency, and increase --stale-ttl proportionally (must remain at least 2× --heartbeat-interval) to avoid false-positive death detection:

--heartbeat-interval 60s \
--stale-ttl 180s \
--reconciliation-interval 120s

Fast recovery priority: Decrease --reconciliation-interval so surviving instances scan for orphaned runs more often. The heartbeat interval and stale TTL control detection speed; the reconciliation interval controls adoption speed:

--heartbeat-interval 15s \
--stale-ttl 45s \
--reconciliation-interval 30s

The tradeoff is more frequent writes and scans against the shared store.

Verify reconciliation is working

If an instance is stopped or killed, its peers should detect the stale heartbeat and adopt its orphaned runs. Check with swamp run history --server against a surviving instance to confirm the runs were picked up.

Serve Flags — full flag and environment variable reference for --heartbeat-interval, --stale-ttl, and --reconciliation-interval
swamp serve — design decisions behind heartbeats, reconciliation, and crash safety
The Run Tracker — per-instance liveness tracking

← PreviousRun as a System Daemon Next →Reload Configuration Without Restart