Skip to main content

WORKER FLEETS

The remote execution model describes how an orchestrator sends work to a single stateless worker. That model works well when each machine is provisioned individually — one token, one worker, one identity. But when the number of workers grows, individual provisioning becomes the bottleneck. Fleets are the answer to that problem.

What a fleet solves

Without fleets, scaling a worker pool means repeating the same ceremony for every machine: create a token, distribute it, connect the worker, track its identity. Each token binds to exactly one machine. If a machine is replaced, a new token must be created. If the pool needs ten workers, ten tokens must be minted and distributed.

This is manageable at small scale but breaks down when workers are ephemeral — containers that come and go with autoscalers, CI runners that exist for one job, spot instances reclaimed without notice. The provisioning overhead dominates the operational cost of running the pool.

A fleet collapses that overhead into a single credential. One fleet token enrolls many machines. The orchestrator handles the rest: naming, labeling, scheduling, and lifecycle management. The operator provisions one token and hands it to an autoscaler, a Deployment, or a Compose service. The individual workers are fungible.

Fleet tokens and enrollment

A fleet token is created with --max-enrollments set to a number greater than one (or unlimited). This changes the token's behavior in two ways.

First, it allows multiple machines to bind to the same token. Each machine presents the token on first connection, and the orchestrator records a separate binding per machine identity. The enrollment allowance is a hard cap — once exhausted, new machines are rejected.

Second, the orchestrator auto-generates a name and label for each enrolled worker. The name follows the pattern <token-name>-<suffix>, where the suffix is derived from the machine's identity. A fleet=<token-name> label is injected automatically. This naming convention exists to solve a practical problem: when ten workers connect with the same token, the operator needs a way to tell them apart in swamp worker list, and workflows need a way to target the pool as a group in placement selectors. The auto-injected label provides that without requiring the operator to configure each worker individually.

The alternative design — requiring the operator to assign names and labels per worker — would reintroduce the per-machine provisioning overhead that fleets exist to eliminate.

Token lifecycle

An enrollment token is time-boxed, and its expiry is a hard deadline. When it passes, every worker bound to the token is removed from the pool at once and cannot enroll again. A time box limits how long a leaked credential is useful, and a deadline that bent for workers already connected would extend that window for the very machines holding the credential. The cost falls on long-lived workers: a worker meant to outlive its token has to be given a new one before the deadline.

Minting a new token under a new name would do that, but the token's name is also the worker's name. A fleet worker is <token-name>-<suffix>, and that name carries the worker's placement targets, its fleet= label, its lane pins and its history. Rotation exists so the credential can change while the name stays put. swamp worker token rotate replaces the secret and the lifetime and keeps the name, the bindings and the enrollment allowance, so from the pool's point of view nothing about the worker changes.

Why the old token overlaps the new one

Delivering a new token to every worker is not instantaneous or guaranteed. Through serve, the new token travels to each connected worker over that worker's own connection, but some workers are offline, some run a Swamp release too old to accept it, and some are connected to a different serve instance than the one that ran the rotation. If the old token stopped working the moment the new one was minted, all of those workers would be cut off by the act meant to keep them running.

So the previous credential, the predecessor, keeps enrolling for a while. Its deadline is its own original expiry, clamped to the new token's: a rotation can shorten how long the old secret is good for, but it never lengthens it. That clamp keeps rotation from becoming a way to extend a credential past the time box it was minted with.

Only one predecessor is kept. A second rotation retires the oldest secret at once instead of stacking another overlap window, and a worker still holding it is cut off. The consequence is that a rotation whose output was lost should not be repeated to recover it — that would retire the original token for every host that has not been updated yet — which is why swamp worker token reveal exists: it reads the current credential back without changing anything.

Where a rotated token lives

A worker that read its token from a file writes the new one back to that file, so it survives a restart. A worker given its token inline, through --token or an environment variable, has nowhere to write it and holds it in memory only. On restart it presents the token it started with, which works until the predecessor's deadline and not after. The report a rotation prints separates these cases because they need different follow-up: the first needs nothing, the second needs the token's source of truth updated.

That is the limit of what rotation reaches. Swamp can deliver a new token to a running worker, but it cannot update a Kubernetes Secret, a Compose file or a CI secret store. Where the token originates outside the worker, the rotation has to be carried back to that source by hand.

Version skew

A worker declares that it can accept a rotated token when it enrolls, as an optional capability alongside its protocol version. Serve only sends a rotation to workers that declared it; an older serve ignores the declaration, and an older worker never makes it. That avoided a protocol version bump: a newer worker still enrolls with an older serve, and an older worker keeps working against a newer one, reported as not rotated.

Rolling serve itself back is the exception. A serve from before rotation knows nothing of predecessors, so it treats the old secret as a mismatch, and every worker still on the old token stops for good. Before rolling serve back past rotation, every worker needs the current token.

Capacity and scheduling

Each worker advertises a number of concurrent dispatch slots via --concurrency. A worker with four slots can execute four steps simultaneously. The orchestrator tracks active dispatches per worker and selects the worker with the most available capacity when multiple workers match a placement selector.

This least-loaded strategy is a deliberate choice over alternatives. Round-robin would distribute evenly by count but ignore that some steps take longer than others, leading to workers sitting idle while others are overloaded. Affinity-based scheduling (preferring the same worker for related steps) would improve cache locality but add complexity and coupling between the scheduler and the workload shape. Least-loaded is simple, observable, and works well for heterogeneous workloads where step duration is unpredictable. When a workflow does need node-local consistency — shared filesystem state, GPU memory, or node-locked licenses — the affinity field pins all placed steps in a scope to the worker selected for the first step, trading parallelism for node-local guarantees on a per-workflow or per-job basis.

Drain and lifecycle policies

Workers need a way to shut down without abandoning in-flight work. Three mechanisms exist for this, each serving a different operational pattern.

--max-dispatches tells a worker to drain after completing a fixed number of dispatches and then exit with code 0. This is designed for ephemeral workers — Kubernetes Jobs, CI runners, spot instances — where each worker should process a bounded amount of work and then be replaced. The orchestrator does not schedule new work to a draining worker.

--idle-timeout tells a worker to drain and exit after a period of continuous inactivity. This supports scale-to-zero patterns where idle workers are expensive and should release their resources.

SIGTERM triggers an immediate drain. The worker finishes any in-flight dispatches (up to the container's grace period), reports results, and exits cleanly. This is the mechanism that container orchestrators — Docker's stop_grace_period, Kubernetes' terminationGracePeriodSeconds — rely on for graceful shutdown. The reason the default grace period recommendation is 300 seconds (five minutes) rather than the typical 30 is that a step in progress may be running a behavior method with substantial computation. A 30-second window risks killing a step mid-execution and losing its results.

All three mechanisms produce exit code 0, signaling to the service manager that the shutdown was intentional and the worker should not be restarted. A non-zero exit indicates a failure — a permanent enrollment rejection, an unexpected error — where a restart may be appropriate.

Elastic queueing

When a workflow step has a placement requirement but no matching worker is connected, the orchestrator faces a choice: fail the step immediately, or queue it and wait.

Swamp queues. The step enters a waiting state, visible through swamp worker queue, and remains there until a matching worker connects or the queue timeout expires. This design was chosen over fail-fast because the most common reason for an empty pool is timing: the worker hasn't connected yet, the autoscaler hasn't provisioned it, the container is still starting. Failing immediately would force the workflow author to build retry logic into every placed workflow. Queueing absorbs the timing gap transparently.

The trade-off is the timeout. Without it, a misconfigured placement selector — requesting a label no worker will ever have — would queue indefinitely. The default timeout is 10 minutes, configurable per-step with queueTimeout or globally with --queue-timeout on the server. Setting the timeout to 0 disables it entirely, which is appropriate when an external system manages the worker lifecycle and is guaranteed to eventually provide a match.

When a worker connects and matches a queued step, dispatch happens immediately — there is no polling interval. The orchestrator evaluates the queue on every enrollment event, so the latency between "worker connects" and "step dispatches" is bounded by the enrollment handshake, not a timer.

Supply and demand

The fleet model creates two observable quantities: supply (connected workers and their available capacity) and demand (pending dispatches waiting for a match). These are exposed through swamp worker list (supply) and swamp worker queue (demand), and are also queryable as data from built-in behaviors for programmatic access.

This visibility matters because scaling decisions — how many workers to run, when to add or remove capacity — depend on the relationship between supply and demand. A queue that grows means supply is insufficient. Workers that are consistently idle mean supply exceeds demand. The orchestrator does not make scaling decisions itself — it provides the data for external systems (autoscalers, operators, monitoring) to make those decisions.

Whose environment a step sees

By default, every dispatched step runs with the orchestrator's environment on top of the worker's, so the orchestrator's credentials and cloud identity reach every worker in the fleet, and its values win over the worker's. A fleet whose workers carry their own identity, such as a per-node-pool IAM role on Kubernetes, loses it to the orchestrator's unless serve is started with dispatch-env-allow. With it set, only the listed names ship and every other variable keeps the worker's value. See The worker environment.

Current limitations

Workers are currently repo-less: they resolve enrollment tokens from CLI flags or environment variables, not from a local Swamp repository. Vault-based token resolution — where a worker could read its enrollment credential from a vault provider the way other Swamp components resolve secrets — is a planned future enhancement. Until then, the first token reaches a worker outside Swamp, through whatever secret management the deployment environment provides (Kubernetes Secrets, Docker secrets, CI secret stores). A rotation reaches connected workers through serve, but carrying it back to those stores is still done by hand.

Further reading