Skip to main content

RUN A FLEET WITH DOCKER COMPOSE

This guide shows you how to deploy a pool of Swamp workers with Docker Compose, using a single fleet token to enroll multiple containers.

Prerequisites

You need a running orchestrator with TLS and token authentication. See Run a Worker in Docker for the orchestrator setup steps (TLS certificate, swamp serve, server token).

Create a fleet token

Create an enrollment token that allows multiple machines to enroll:

swamp worker token create compose-fleet --duration 7d --max-enrollments 20

Save the token value. Each container that connects with this token receives an auto-generated name (compose-fleet-<suffix>) and a fleet=compose-fleet label.

Write the Compose file

services:
  worker:
    image: swamp-worker
    environment:
      SWAMP_ORCHESTRATOR_URL: wss://host.docker.internal:9090
      SWAMP_SERVER_TOKEN: admin.<secret>
      SWAMP_WORKER_TOKEN: compose-fleet.<secret>
      SWAMP_WORKER_CONCURRENCY: "4"
      DENO_CERT: /certs/cert.pem
    volumes:
      - ./cert.pem:/certs/cert.pem:ro
    command: ["worker", "connect"]
    stop_grace_period: 5m
    restart: on-failure

stop_grace_period: 5m gives in-flight steps five minutes to complete when Docker sends SIGTERM during docker compose down or scaling events. If a step takes longer than the grace period, Docker force-kills the container and the step is lost.

restart: on-failure restarts a worker that crashes but does not restart one that exits cleanly (exit code 0). A clean exit happens when --max-dispatches or --idle-timeout triggers a drain.

SWAMP_WORKER_CONCURRENCY sets the number of concurrent dispatch slots per container. Adjust based on the workload — CPU-bound steps benefit from fewer slots; I/O-bound steps can handle more.

Scale the fleet

Start with one worker, then scale up:

docker compose up -d
docker compose up -d --scale worker=5

Verify enrollment:

swamp worker list

Each worker appears with its auto-generated name and the fleet=compose-fleet label.

To scale down, Docker sends SIGTERM to the excess containers. Each drains its in-flight work before exiting:

docker compose up -d --scale worker=2

Run a parallel workload

Create a workflow with a forEach that fans work across the fleet:

jobs:
  - name: parallel-work
    forEach:
      items: ["a", "b", "c", "d", "e"]
    steps:
      - name: process
        task:
          type: model_method
          modelIdOrName: "processor-${{ self.item }}"
          methodName: run
        labels:
          fleet: compose-fleet
swamp workflow run fan-out

Steps dispatch to available workers across the fleet. The orchestrator selects the worker with the most available capacity for each step.

Per-model lock: steps that write to the same model definition serialize on the definition's datastore lock. If the forEach items all target the same modelIdOrName, they run sequentially regardless of fleet size. Use distinct model names (as above, with ${{ self.item }}) for true parallelism. See Workflow Placement for scheduling details.

Add lifecycle policies

To run workers that process a fixed number of steps and exit:

environment:
  SWAMP_WORKER_MAX_DISPATCHES: "10"

To run workers that exit after 10 minutes of inactivity:

environment:
  SWAMP_WORKER_IDLE_TIMEOUT: 10m

Both produce exit code 0, so restart: on-failure does not restart the container.

← PreviousRun a Worker in DockerNext →Run a Fleet on Kubernetes