RESTART SWAMP SERVE WITHOUT DROPPING WORK
This guide shows you how to restart swamp serve, for an upgrade, a new
certificate or a setting that swamp serve reload cannot change, without losing
runs or failing the requests that arrive during the restart.
A planned restart hands runs over: serve lets running steps finish, parks each
run at a step boundary, and the next process continues it under the same run id.
The CLI, @swamp-club/swamp-lib and workers retry through the restart. What
differs between the methods below is whether connections are refused while no
process is listening. See
Startup and shutdown
for every refusal and retry window.
| Method | Refused connections | Where |
|---|---|---|
| Restart in place | Until the new process is ready | Any OS |
| Start the new process beside the old one | None | Linux, not on a v3 datastore |
| Restart a team one host at a time | None behind a load balancer | A v3 team |
Before you restart
- Set
--shutdown-drain-timeout(default30s) longer than the steps you expect to be running. A step still running at the timeout is aborted, and its run continues only if that step has aguard. - Keep the drain timeout above
0. At0runs are aborted, not handed over. - Give your process supervisor time to wait for the drain: a stop timeout above the drain timeout plus about 25 seconds.
Restart in place
Stop serve with SIGTERM and start it again. On any OS this is what your
supervisor does on restart:
sudo systemctl restart swamp-serveServe logs the drain as it stops:
Shutting down, draining in-flight runs for up to 30s...
Draining 1 active run(s)...and the run it parked as it starts again:
Run "1a031b49-3e30-45eb-8bae-27f86982e111" of workflow "slow-deploy", interrupted by "server_restart", is continued automatically
Continued 1 interrupted run(s) automatically
Startup complete — /ready is now serving 200A client attached to the run with --server reconnects and streams the rest of
the run:
Connection dropped, reconnecting for up to 120s...A run started while serve is down or draining is retried:
serve at http://127.0.0.1:19091 is restarting (draining); retrying for up to 120sWebhook senders do not retry. To fail no delivery, put a proxy in front of serve
that retries a 503 or a refused connection, or use one of the methods below.
Start the new process beside the old one
On Linux, --reuse-port lets two serve processes listen on the same port. Serve
opens the port only once startup completes, so the new process takes no
connection before it is ready, and the old one takes none after it closes its
listener.
This does not work on a v3 datastore: a second process there becomes a follower of the first. Restart a v3 team one host at a time instead.
Start every serve with
--reuse-port. A process started without it holds the port alone, and an overlapping start is refused with--reuse-port: cannot listen on …. On macOS and Windows every second process is refused, and serve warns at startup:--reuse-port: only Linux lets two processes share a port, so on "darwin" a second serve cannot start beside this oneError: --reuse-port: cannot listen on 127.0.0.1:9090 (Address already in use (os error 48)); a process listening there without --reuse-port blocks an overlapping startRun serve from a systemd template unit, so two instances can run at once. Save it as
/etc/systemd/system/swamp-serve@.service:[Unit] Description=swamp serve (%i) After=network-online.target [Service] User=swamp ExecStart=/usr/local/bin/swamp serve --repo-dir /srv/swamp --reuse-port KillSignal=SIGTERM TimeoutStopSec=60 [Install] WantedBy=multi-user.targetTimeoutStopSeccovers the default 30 second drain plus teardown. Add the environment and credentials your serve needs, as in Run as a System Daemon. Start the first instance:sudo systemctl daemon-reload sudo systemctl enable --now swamp-serve@a
Start the new instance and wait until it is ready. Both instances answer on the same port, so tell them apart by the
instanceIdin/ready:old=$(curl -s http://127.0.0.1:9090/ready | jq -r .instanceId) sudo systemctl start swamp-serve@b until [ "$(curl -s http://127.0.0.1:9090/ready | jq -r .instanceId)" != "$old" ]; do sleep 1; done
Repeat the
curlif it keeps reaching the old instance: the kernel spreads connections across both.Stop the old instance:
sudo systemctl disable --now swamp-serve@a
It drains and parks its runs, and the new instance continues them. A run still queued on the old process is replayed by the new one once the old one has stopped.
For the next restart, swap a and b.
Check that runs were continued
swamp run history --server https://swamp.example.comA handed-over run shows interrupted (server_restart) while it waits, and
[CONTINUED] once it has finished:
STATUS KIND NAME ID AGE INITIATED BY PID HOST
completed [CONTINUED] workflow slow-deploy 1a031b49 33s ghost 16548 my-hostswamp workflow history get <run-id> names the restart:
Continued automatically after server_restart at 2026-11-19 17:19:47 UTCA run that stays interrupted was not continued. Its history says why, with the
swamp workflow recover command that decides it. A workflow with
autoContinue: false always waits
for that command.
Related
- Serve Flags — the shutdown sequence, refusals and client retry windows
- Run a Multi-Instance Deployment — restart a team one host at a time
- Deploy swamp serve on Kubernetes — probes and grace periods for a rollout
- Set Up TLS for swamp serve — rotate serve's certificate with a restart
- Workflows — when an interrupted run is continued