Skip to main content

RUN A MULTI-INSTANCE DEPLOYMENT

This guide shows you how to run swamp serve in a multi-instance deployment where two or more instances share a datastore behind a load balancer.

Prerequisites

  • A swamp-club.com credential for every instance: a collective token as SWAMP_API_KEY plus its SWAMP_SIGNIN_TOKEN, so instances start while swamp-club.com is unreachable — see Run Swamp in CI
  • A shared datastore (e.g. S3) accessible by all instances
  • A load balancer or reverse proxy distributing client connections
  • TLS certificates and an authentication mode (token or oauth) — required for off-loopback binding
  • For OAuth mode: SWAMP_API_KEY with oauth:manage + serve:* scopes for headless bootstrap — see Deploy with Headless OAuth
  • For shared config across instances: enable managed config so behavior definitions, workflows, and vault configs propagate automatically — see Enable Managed Config

HA auto-detection

HA mode is detected automatically based on the datastore configuration. When the datastore supports a control plane (e.g. S3), runs are automatically detached from the client connection and reconciliation is enabled — no flag required:

swamp serve \
  --host 0.0.0.0 \
  --auth-mode token \
  --cert-file /etc/swamp/tls/cert.pem \
  --key-file /etc/swamp/tls/key.pem

Note

The --detach-runs flag is deprecated. It is still accepted for backwards compatibility but has no effect — HA mode is determined entirely by the datastore.

Grant, revoke and group changes made through one instance reach the others within one --datastore-poll-interval. With --grant-reload manual, reload only the instance the change went through. See Reload the policy snapshot.

Take writes on every node of a v3 team

On a v3 datastore one node leads and the others follow. Every node behind the load balancer takes writes: a follower forwards each write to the leader with the caller's own credentials, and the leader authenticates and authorizes the caller. Followers follow the leader's commit feed, so a write reaches them in about a second, and grant changes made on the leader apply on every node as they land.

  1. Advertise each node's address. Start every node with --advertise-address, its own https origin as the other nodes reach it. Add --ca-cert when the nodes' certificates come from a private CA:

    swamp serve \
  --host 0.0.0.0 \
  --auth-mode token \
  --cert-file /etc/swamp/tls/cert.pem \
  --key-file /etc/swamp/tls/key.pem \
  --advertise-address https://node-1.internal:9090 \
  --ca-cert /etc/swamp/tls/internal-ca.pem \
  --trust-proxy

    In serve.yaml these are advertise-address and tls.ca-cert. An address with a path, query, credentials or wildcard host is refused:

    Error: The advertise address must be this node's own http(s) origin, as the team's other serve nodes reach it, e.g. https://node-1.internal:9090: no path, query, credentials or wildcard host

    A node without an advertised address, bound to a wildcard or loopback host, cannot be forwarded to.

  2. Trust the load balancer. Run every node with --trust-proxy, so each node limits and audits callers by the address in X-Forwarded-For rather than the load balancer's. Forwards from a follower to the leader need nothing extra: the leader limits each forwarded caller as itself.

  3. Keep the token-secrets key outside the datastore. With the default co-located key, anyone who can read and write the datastore can sign a leader record of their own, and followers then send every forwarded caller's credentials to its address. A team-mode node says so in its start-up warning. Move the key into a vault: Keep the token-secrets key outside the datastore.

  4. Give every node the same webhook config. A follower forwards each webhook delivery to the leader, which verifies it with its own routes and secrets. See Webhooks.

Retry a write lost during a leader change

A forwarded write that reached the leader but got no reply, because the leader changed or stopped answering, fails with serve_not_leader and inFlight: true. The follower never retries it. The write may already have run: check its outcome before you retry. With --json:

{
  "error": "This swamp serve node forwarded the request to the leader, which changed or stopped answering before it replied. The request may already have run: check its outcome before retrying.",
  "code": "serve_not_leader",
  "inFlight": true
}

A serve_not_leader error without inFlight changed nothing; retry it.

Tune the timing constants

Three flags control how quickly a dead instance is detected and its runs recovered. All three only take effect with a control-plane-capable datastore.

swamp serve \
  --heartbeat-interval 30s \
  --stale-ttl 90s \
  --reconciliation-interval 60s \
  # ... other flags

Adjust these based on your deployment's characteristics:

Low-latency datastore (local SSD, same-region S3): The defaults work well. A dead instance is detected within ~90s and its runs reconciled within ~150s.

High-latency or unreliable datastore (cross-region S3, high-jitter networks): Increase --heartbeat-interval to allow for write latency, and increase --stale-ttl proportionally (must remain at least 2× --heartbeat-interval) to avoid false-positive death detection:

--heartbeat-interval 60s \
--stale-ttl 180s \
--reconciliation-interval 120s

Fast recovery priority: Decrease --reconciliation-interval so surviving instances scan for orphaned runs more often. The heartbeat interval and stale TTL control detection speed; the reconciliation interval controls adoption speed:

--heartbeat-interval 15s \
--stale-ttl 45s \
--reconciliation-interval 30s

The tradeoff is more frequent writes and scans against the shared store.

Verify reconciliation is working

If an instance is stopped or killed, its peers should detect the stale heartbeat and adopt its orphaned runs. Check with swamp run history --server against a surviving instance to confirm the runs were picked up.

← PreviousRun as a System DaemonNext →Reload Configuration Without Restart