WORKER FLEETS
The remote execution model describes how an orchestrator sends work to a single stateless worker. That model works well when each machine is provisioned individually — one token, one worker, one identity. But when the number of workers grows, individual provisioning becomes the bottleneck. Fleets are the answer to that problem.
What a fleet solves
Without fleets, scaling a worker pool means repeating the same ceremony for every machine: create a token, distribute it, connect the worker, track its identity. Each token binds to exactly one machine. If a machine is replaced, a new token must be created. If the pool needs ten workers, ten tokens must be minted and distributed.
This is manageable at small scale but breaks down when workers are ephemeral — containers that come and go with autoscalers, CI runners that exist for one job, spot instances reclaimed without notice. The provisioning overhead dominates the operational cost of running the pool.
A fleet collapses that overhead into a single credential. One fleet token enrolls many machines. The orchestrator handles the rest: naming, labeling, scheduling, and lifecycle management. The operator provisions one token and hands it to an autoscaler, a Deployment, or a Compose service. The individual workers are fungible.
Fleet tokens and enrollment
A fleet token is created with --max-enrollments set to a number greater than
one (or unlimited). This changes the token's behavior in two ways.
First, it allows multiple machines to bind to the same token. Each machine presents the token on first connection, and the orchestrator records a separate binding per machine identity. The enrollment allowance is a hard cap — once exhausted, new machines are rejected.
Second, the orchestrator auto-generates a name and label for each enrolled
worker. The name follows the pattern <token-name>-<suffix>, where the suffix
is derived from the machine's identity. A fleet=<token-name> label is injected
automatically. This naming convention exists to solve a practical problem: when
ten workers connect with the same token, the operator needs a way to tell them
apart in swamp worker list, and workflows need a way to target the pool as a
group in
placement selectors.
The auto-injected label provides that without requiring the operator to
configure each worker individually.
The alternative design — requiring the operator to assign names and labels per worker — would reintroduce the per-machine provisioning overhead that fleets exist to eliminate.
Token lifecycle
An enrollment token is time-boxed, and its expiry is a hard deadline. When it passes, every worker bound to the token is removed from the pool at once and cannot enroll again. A time box limits how long a leaked credential is useful, and a deadline that bent for workers already connected would extend that window for the very machines holding the credential. The cost falls on long-lived workers: a worker meant to outlive its token has to be given a new one before the deadline.
Minting a new token under a new name would do that, but the token's name is also
the worker's name. A fleet worker is <token-name>-<suffix>, and that name
carries the worker's placement targets, its fleet= label, its lane pins and
its history. Rotation exists so the credential can change while the name stays
put. swamp worker token rotate replaces the secret and the lifetime and keeps
the name, the bindings and the enrollment allowance, so from the pool's point of
view nothing about the worker changes.
Why the old token overlaps the new one
Delivering a new token to every worker is not instantaneous or guaranteed. Through serve, the new token travels to each connected worker over that worker's own connection, but some workers are offline, some run a Swamp release too old to accept it, and some are connected to a different serve instance than the one that ran the rotation. If the old token stopped working the moment the new one was minted, all of those workers would be cut off by the act meant to keep them running.
So the previous credential, the predecessor, keeps enrolling for a while. Its deadline is its own original expiry, clamped to the new token's: a rotation can shorten how long the old secret is good for, but it never lengthens it. That clamp keeps rotation from becoming a way to extend a credential past the time box it was minted with.
Only one predecessor is kept. A second rotation retires the oldest secret at
once instead of stacking another overlap window, and a worker still holding it
is cut off. The consequence is that a rotation whose output was lost should not
be repeated to recover it — that would retire the original token for every host
that has not been updated yet — which is why swamp worker token reveal exists:
it reads the current credential back without changing anything.
Where a rotated token lives
A worker that read its token from a file writes the new one back to that file,
so it survives a restart. A worker given its token inline, through --token or
an environment variable, has nowhere to write it and holds it in memory only. On
restart it presents the token it started with, which works until the
predecessor's deadline and not after. The report a rotation prints separates
these cases because they need different follow-up: the first needs nothing, the
second needs the token's source of truth updated.
That is the limit of what rotation reaches. Swamp can deliver a new token to a running worker, but it cannot update a Kubernetes Secret, a Compose file or a CI secret store. Where the token originates outside the worker, the rotation has to be carried back to that source by hand.
Version skew
A worker declares that it can accept a rotated token when it enrolls, as an optional capability alongside its protocol version. Serve only sends a rotation to workers that declared it; an older serve ignores the declaration, and an older worker never makes it. That avoided a protocol version bump: a newer worker still enrolls with an older serve, and an older worker keeps working against a newer one, reported as not rotated.
Rolling serve itself back is the exception. A serve from before rotation knows nothing of predecessors, so it treats the old secret as a mismatch, and every worker still on the old token stops for good. Before rolling serve back past rotation, every worker needs the current token.
Capacity and scheduling
Each worker advertises a number of concurrent dispatch slots via
--concurrency. A worker with four slots can execute four steps simultaneously.
The orchestrator tracks active dispatches per worker and selects the worker with
the most available capacity when multiple workers match a placement selector.
This least-loaded strategy is a deliberate choice over alternatives. Round-robin
would distribute evenly by count but ignore that some steps take longer than
others, leading to workers sitting idle while others are overloaded.
Affinity-based scheduling (preferring the same worker for related steps) would
improve cache locality but add complexity and coupling between the scheduler and
the workload shape. Least-loaded is simple, observable, and works well for
heterogeneous workloads where step duration is unpredictable. When a workflow
does need node-local consistency — shared filesystem state, GPU memory, or
node-locked licenses — the
affinity field
pins all placed steps in a scope to the worker selected for the first step,
trading parallelism for node-local guarantees on a per-workflow or per-job
basis.
Drain and lifecycle policies
Workers need a way to shut down without abandoning in-flight work. Three mechanisms exist for this, each serving a different operational pattern.
--max-dispatches tells a worker to drain after completing a fixed number of
dispatches and then exit with code 0. This is designed for ephemeral workers —
Kubernetes Jobs, CI runners, spot instances — where each worker should process a
bounded amount of work and then be replaced. The orchestrator does not schedule
new work to a draining worker.
--idle-timeout tells a worker to drain and exit after a period of continuous
inactivity. This supports scale-to-zero patterns where idle workers are
expensive and should release their resources.
SIGTERM triggers an immediate drain. The worker finishes any in-flight
dispatches (up to the container's grace period), reports results, and exits
cleanly. This is the mechanism that container orchestrators — Docker's
stop_grace_period, Kubernetes' terminationGracePeriodSeconds — rely on for
graceful shutdown. The reason the default grace period recommendation is 300
seconds (five minutes) rather than the typical 30 is that a step in progress may
be running a behavior method with substantial computation. A 30-second window
risks killing a step mid-execution and losing its results.
All three mechanisms produce exit code 0, signaling to the service manager that the shutdown was intentional and the worker should not be restarted. A non-zero exit indicates a failure — a permanent enrollment rejection, an unexpected error — where a restart may be appropriate.
Elastic queueing
When a workflow step has a placement requirement but no matching worker is connected, the orchestrator faces a choice: fail the step immediately, or queue it and wait.
Swamp queues. The step enters a waiting state, visible through
swamp worker queue, and remains there until a matching worker connects or the
queue timeout expires. This design was chosen over fail-fast because the most
common reason for an empty pool is timing: the worker hasn't connected yet, the
autoscaler hasn't provisioned it, the container is still starting. Failing
immediately would force the workflow author to build retry logic into every
placed workflow. Queueing absorbs the timing gap transparently.
The trade-off is the timeout. Without it, a misconfigured placement selector —
requesting a label no worker will ever have — would queue indefinitely. The
default timeout is 10 minutes, configurable per-step with queueTimeout or
globally with --queue-timeout on the server. Setting the timeout to 0 disables
it entirely, which is appropriate when an external system manages the worker
lifecycle and is guaranteed to eventually provide a match.
When a worker connects and matches a queued step, dispatch happens immediately — there is no polling interval. The orchestrator evaluates the queue on every enrollment event, so the latency between "worker connects" and "step dispatches" is bounded by the enrollment handshake, not a timer.
Supply and demand
The fleet model creates two observable quantities: supply (connected workers and
their available capacity) and demand (pending dispatches waiting for a match).
These are exposed through swamp worker list (supply) and swamp worker queue
(demand), and are also queryable as data from built-in behaviors for
programmatic access.
This visibility matters because scaling decisions — how many workers to run, when to add or remove capacity — depend on the relationship between supply and demand. A queue that grows means supply is insufficient. Workers that are consistently idle mean supply exceeds demand. The orchestrator does not make scaling decisions itself — it provides the data for external systems (autoscalers, operators, monitoring) to make those decisions.
Whose environment a step sees
By default, every dispatched step runs with the orchestrator's environment on
top of the worker's, so the orchestrator's credentials and cloud identity reach
every worker in the fleet, and its values win over the worker's. A fleet whose
workers carry their own identity, such as a per-node-pool IAM role on
Kubernetes, loses it to the orchestrator's unless serve is started with
dispatch-env-allow.
With it set, only the listed names ship and every other variable keeps the
worker's value. See
The worker environment.
Current limitations
Workers are currently repo-less: they resolve enrollment tokens from CLI flags or environment variables, not from a local Swamp repository. Vault-based token resolution — where a worker could read its enrollment credential from a vault provider the way other Swamp components resolve secrets — is a planned future enhancement. Until then, the first token reaches a worker outside Swamp, through whatever secret management the deployment environment provides (Kubernetes Secrets, Docker secrets, CI secret stores). A rotation reaches connected workers through serve, but carrying it back to those stores is still done by hand.
Further reading
- Worker Fleets how-to guides — practical guides for Docker, Docker Compose, Kubernetes, and scale-from-zero
- Rotate a Worker Enrollment Token — keep workers connected past their token's expiry
- Remote Execution — the underlying orchestrator-worker model
- Enrollment Tokens — token lifecycle, machine binding, fleet token details
- Worker Commands — CLI reference for fleet management
- Workflow Placement — step placement fields and scheduling behavior