Worker Fleets

The remote execution model describes how an orchestrator sends work to a single stateless worker. That model works well when each machine is provisioned individually — one token, one worker, one identity. But when the number of workers grows, individual provisioning becomes the bottleneck. Fleets are the answer to that problem.

What a fleet solves

Without fleets, scaling a worker pool means repeating the same ceremony for every machine: create a token, distribute it, connect the worker, track its identity. Each token binds to exactly one machine. If a machine is replaced, a new token must be created. If the pool needs ten workers, ten tokens must be minted and distributed.

This is manageable at small scale but breaks down when workers are ephemeral — containers that come and go with autoscalers, CI runners that exist for one job, spot instances reclaimed without notice. The provisioning overhead dominates the operational cost of running the pool.

A fleet collapses that overhead into a single credential. One fleet token enrolls many machines. The orchestrator handles the rest: naming, labeling, scheduling, and lifecycle management. The operator provisions one token and hands it to an autoscaler, a Deployment, or a Compose service. The individual workers are fungible.

Fleet tokens and enrollment

A fleet token is created with --max-enrollments set to a number greater than one (or unlimited). This changes the token's behavior in two ways.

First, it allows multiple machines to bind to the same token. Each machine presents the token on first connection, and the orchestrator records a separate binding per machine identity. The enrollment allowance is a hard cap — once exhausted, new machines are rejected.

Second, the orchestrator auto-generates a name and label for each enrolled worker. The name follows the pattern <token-name>-<suffix>, where the suffix is derived from the machine's identity. A fleet=<token-name> label is injected automatically. This naming convention exists to solve a practical problem: when ten workers connect with the same token, the operator needs a way to tell them apart in swamp worker list, and workflows need a way to target the pool as a group in placement selectors. The auto-injected label provides that without requiring the operator to configure each worker individually.

The alternative design — requiring the operator to assign names and labels per worker — would reintroduce the per-machine provisioning overhead that fleets exist to eliminate.

Capacity and scheduling

Each worker advertises a number of concurrent dispatch slots via --concurrency. A worker with four slots can execute four steps simultaneously. The orchestrator tracks active dispatches per worker and selects the worker with the most available capacity when multiple workers match a placement selector.

This least-loaded strategy is a deliberate choice over alternatives. Round-robin would distribute evenly by count but ignore that some steps take longer than others, leading to workers sitting idle while others are overloaded. Affinity-based scheduling (preferring the same worker for related steps) would improve cache locality but add complexity and coupling between the scheduler and the workload shape. Least-loaded is simple, observable, and works well for heterogeneous workloads where step duration is unpredictable.

Drain and lifecycle policies

Workers need a way to shut down without abandoning in-flight work. Three mechanisms exist for this, each serving a different operational pattern.

--max-dispatches tells a worker to drain after completing a fixed number of dispatches and then exit with code 0. This is designed for ephemeral workers — Kubernetes Jobs, CI runners, spot instances — where each worker should process a bounded amount of work and then be replaced. The orchestrator does not schedule new work to a draining worker.

--idle-timeout tells a worker to drain and exit after a period of continuous inactivity. This supports scale-to-zero patterns where idle workers are expensive and should release their resources.

SIGTERM triggers an immediate drain. The worker finishes any in-flight dispatches (up to the container's grace period), reports results, and exits cleanly. This is the mechanism that container orchestrators — Docker's stop_grace_period, Kubernetes' terminationGracePeriodSeconds — rely on for graceful shutdown. The reason the default grace period recommendation is 300 seconds (five minutes) rather than the typical 30 is that a step in progress may be running a model method with substantial computation. A 30-second window risks killing a step mid-execution and losing its results.

All three mechanisms produce exit code 0, signaling to the service manager that the shutdown was intentional and the worker should not be restarted. A non-zero exit indicates a failure — a permanent enrollment rejection, an unexpected error — where a restart may be appropriate.

Elastic queueing

When a workflow step has a placement requirement but no matching worker is connected, the orchestrator faces a choice: fail the step immediately, or queue it and wait.

Swamp queues. The step enters a waiting state, visible through swamp worker queue, and remains there until a matching worker connects or the queue timeout expires. This design was chosen over fail-fast because the most common reason for an empty pool is timing: the worker hasn't connected yet, the autoscaler hasn't provisioned it, the container is still starting. Failing immediately would force the workflow author to build retry logic into every placed workflow. Queueing absorbs the timing gap transparently.

The trade-off is the timeout. Without it, a misconfigured placement selector — requesting a label no worker will ever have — would queue indefinitely. The default timeout is 10 minutes, configurable per-step with queueTimeout or globally with --queue-timeout on the server. Setting the timeout to 0 disables it entirely, which is appropriate when an external system manages the worker lifecycle and is guaranteed to eventually provide a match.

When a worker connects and matches a queued step, dispatch happens immediately — there is no polling interval. The orchestrator evaluates the queue on every enrollment event, so the latency between "worker connects" and "step dispatches" is bounded by the enrollment handshake, not a timer.

Supply and demand

The fleet model creates two observable quantities: supply (connected workers and their available capacity) and demand (pending dispatches waiting for a match). These are exposed through swamp worker list (supply) and swamp worker queue (demand), and are also queryable as built-in data models for programmatic access.

This visibility matters because scaling decisions — how many workers to run, when to add or remove capacity — depend on the relationship between supply and demand. A queue that grows means supply is insufficient. Workers that are consistently idle mean supply exceeds demand. The orchestrator does not make scaling decisions itself — it provides the data for external systems (autoscalers, operators, monitoring) to make those decisions.

Current limitations

Workers are currently repo-less: they resolve enrollment tokens from CLI flags or environment variables, not from a local Swamp repository. Vault-based token resolution — where a worker could read its enrollment credential from a vault provider the way other Swamp components resolve secrets — is a planned future enhancement. Until then, token distribution happens outside Swamp, through whatever secret management the deployment environment provides (Kubernetes Secrets, Docker secrets, CI secret stores).