Skip to main content

RESTART SWAMP SERVE WITHOUT DROPPING WORK

This guide shows you how to restart swamp serve, for an upgrade, a new certificate or a setting that swamp serve reload cannot change, without losing runs or failing the requests that arrive during the restart.

A planned restart hands runs over: serve lets running steps finish, parks each run at a step boundary, and the next process continues it under the same run id. The CLI, @swamp-club/swamp-lib and workers retry through the restart. What differs between the methods below is whether connections are refused while no process is listening. See Startup and shutdown for every refusal and retry window.

Method Refused connections Where
Restart in place Until the new process is ready Any OS
Start the new process beside the old one None Linux, not on a v3 datastore
Restart a team one host at a time None behind a load balancer A v3 team

Before you restart

  • Set --shutdown-drain-timeout (default 30s) longer than the steps you expect to be running. A step still running at the timeout is aborted, and its run continues only if that step has a guard.
  • Keep the drain timeout above 0. At 0 runs are aborted, not handed over.
  • Give your process supervisor time to wait for the drain: a stop timeout above the drain timeout plus about 25 seconds.

Restart in place

Stop serve with SIGTERM and start it again. On any OS this is what your supervisor does on restart:

sudo systemctl restart swamp-serve

Serve logs the drain as it stops:

Shutting down, draining in-flight runs for up to 30s...
Draining 1 active run(s)...

and the run it parked as it starts again:

Run "1a031b49-3e30-45eb-8bae-27f86982e111" of workflow "slow-deploy", interrupted by "server_restart", is continued automatically
Continued 1 interrupted run(s) automatically
Startup complete — /ready is now serving 200

A client attached to the run with --server reconnects and streams the rest of the run:

Connection dropped, reconnecting for up to 120s...

A run started while serve is down or draining is retried:

serve at http://127.0.0.1:19091 is restarting (draining); retrying for up to 120s

Webhook senders do not retry. To fail no delivery, put a proxy in front of serve that retries a 503 or a refused connection, or use one of the methods below.

Start the new process beside the old one

On Linux, --reuse-port lets two serve processes listen on the same port. Serve opens the port only once startup completes, so the new process takes no connection before it is ready, and the old one takes none after it closes its listener.

This does not work on a v3 datastore: a second process there becomes a follower of the first. Restart a v3 team one host at a time instead.

  1. Start every serve with --reuse-port. A process started without it holds the port alone, and an overlapping start is refused with --reuse-port: cannot listen on …. On macOS and Windows every second process is refused, and serve warns at startup:

    --reuse-port: only Linux lets two processes share a port, so on "darwin" a second serve cannot start beside this one
    Error: --reuse-port: cannot listen on 127.0.0.1:9090 (Address already in use (os error 48)); a process listening there without --reuse-port blocks an overlapping start
  2. Run serve from a systemd template unit, so two instances can run at once. Save it as /etc/systemd/system/swamp-serve@.service:

    [Unit]
    Description=swamp serve (%i)
    After=network-online.target
    
    [Service]
    User=swamp
    ExecStart=/usr/local/bin/swamp serve --repo-dir /srv/swamp --reuse-port
    KillSignal=SIGTERM
    TimeoutStopSec=60
    
    [Install]
    WantedBy=multi-user.target

    TimeoutStopSec covers the default 30 second drain plus teardown. Add the environment and credentials your serve needs, as in Run as a System Daemon. Start the first instance:

    sudo systemctl daemon-reload
    sudo systemctl enable --now swamp-serve@a
  3. Start the new instance and wait until it is ready. Both instances answer on the same port, so tell them apart by the instanceId in /ready:

    old=$(curl -s http://127.0.0.1:9090/ready | jq -r .instanceId)
    sudo systemctl start swamp-serve@b
    until [ "$(curl -s http://127.0.0.1:9090/ready | jq -r .instanceId)" != "$old" ]; do sleep 1; done

    Repeat the curl if it keeps reaching the old instance: the kernel spreads connections across both.

  4. Stop the old instance:

    sudo systemctl disable --now swamp-serve@a

    It drains and parks its runs, and the new instance continues them. A run still queued on the old process is replayed by the new one once the old one has stopped.

For the next restart, swap a and b.

Check that runs were continued

swamp run history --server https://swamp.example.com

A handed-over run shows interrupted (server_restart) while it waits, and [CONTINUED] once it has finished:

STATUS                 KIND      NAME                   ID        AGE  INITIATED BY  PID    HOST
completed [CONTINUED]  workflow  slow-deploy            1a031b49  33s  ghost         16548  my-host

swamp workflow history get <run-id> names the restart:

Continued automatically after server_restart at 2026-11-19 17:19:47 UTC

A run that stays interrupted was not continued. Its history says why, with the swamp workflow recover command that decides it. A workflow with autoContinue: false always waits for that command.