WORKER COMMANDS
CLI reference for managing remote execution workers and the Swamp server. This
page covers token lifecycle commands, worker enrollment, pool inspection, queue
visibility, and the swamp serve entry point.
For enrollment token lifecycle details (machine binding, expiry, state transitions), see Enrollment Tokens. For step-level placement fields that route work to specific workers, see Workflow Placement.
Global flags
All commands in this section accept the standard global options. These are documented once here rather than repeated per command.
| Flag | Description |
|---|---|
--json |
Emit structured JSON instead of human-readable output |
--log <path> |
Write log output to the given file |
--log-level <lvl> |
Set log verbosity (debug, info, warn, error) |
-q |
Quiet mode — suppress non-essential output |
-v |
Verbose mode — increase output detail |
--no-telemetry |
Disable telemetry for this invocation |
--show-properties |
Print resolved configuration properties |
--no-color |
Disable colored output |
swamp worker token create
Mint a named worker enrollment token. The plaintext is shown exactly once in
<name>.<secret> format.
swamp worker token create <name> --duration <duration> [--vault <vault>] [--max-enrollments <n>]Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--duration <duration> |
Yes | Token lifetime (e.g. 30m, 1h, 24h, 7d). Hard deadline: the enrolled worker is disconnected when it elapses. |
-- |
--vault <vault> |
No | Vault that stores the token plaintext | Sole configured vault |
--max-enrollments <n> |
No | Maximum machines this token can enroll (positive integer or "unlimited"). Fleet tokens name workers as <token>-<suffix> and auto-inject a fleet=<token> label. |
1 |
Example
$ swamp worker token create ci-runner --duration 24hToken: ci-runner
Expires: 2026-06-16T10:34:17.833Z
Vault: sandbox-vault (key worker-token-ci-runner)
ci-runner.9ce100bf00607d0edc3f231d6f7d85d028ba820b238fda113469cf0fdf7bfa97
This token is shown once and will not be displayed again — store it now.The token value (ci-runner.9ce100bf...) is the credential passed to
swamp worker connect --token. The name prefix before the dot identifies the
token; the suffix after the dot is the secret. See
Enrollment Tokens for
the full token lifecycle.
Fleet token example
$ swamp worker token create gpu-fleet --duration 7d --max-enrollments 10Token: gpu-fleet
Expires: 2026-07-11T00:00:00.000Z
Vault: sandbox-vault (key worker-token-gpu-fleet)
Max enrollments: 10
gpu-fleet.a4e29c1f00817b3eac5f642d7e8c96b139ca931b349fea224580de1fef8cbaa8
This token is shown once and will not be displayed again — store it now.swamp worker token list
List worker enrollment tokens showing recorded vs effective (display) state, expiry, and enrollments.
swamp worker token listTable output
$ swamp worker token listNAME STATE EXPIRES ENROLLMENTS
ci-runner enrolled 2026-06-16T10:34:17.833Z machine-42
gpu-fleet enrolled 2026-07-11T00:00:00.000Z 3 / 10Single-enrollment tokens show the bound machine ID (e.g. machine-42). Fleet
tokens show used / allowance (e.g. 3 / 10, 8 / unlimited).
JSON output
$ swamp worker token list --json[
{
"name": "ci-runner",
"state": "enrolled",
"effectiveState": "enrolled",
"createdAt": "2026-06-15T10:34:17.833Z",
"expiresAt": "2026-06-16T10:34:17.833Z",
"maxEnrollments": 1,
"bindingCount": 1,
"bindings": [
{ "machineId": "machine-42", "enrolledAt": "2026-06-15T10:34:30.000Z" }
],
"vaultName": "sandbox-vault",
"secretKey": "worker-token-ci-runner"
}
]JSON fields
| Field | Type | Description |
|---|---|---|
name |
string |
Token name as provided at creation |
state |
string |
Recorded state (unused, enrolled, revoked) |
effectiveState |
string |
Display state accounting for expiry (may differ from state) |
createdAt |
string |
ISO 8601 creation timestamp |
expiresAt |
string |
ISO 8601 expiry timestamp |
maxEnrollments |
number |
Maximum machines this token can enroll (1 for single-enrollment tokens) |
bindingCount |
number |
Number of machines currently bound to this token |
bindings |
array |
List of bound machines, each with machineId (string) and enrolledAt (ISO 8601) |
vaultName |
string |
Vault storing the token plaintext |
secretKey |
string |
Key under which the token is stored in the vault |
swamp worker token revoke
Invalidate a worker enrollment token before it expires.
swamp worker token revoke <name>Example
$ swamp worker token revoke ci-runner✓ Token ci-runner revoked.
Revoked at: 2026-06-15T10:34:39.736ZAfter revocation, swamp worker token list reflects the updated state:
NAME STATE EXPIRES ENROLLMENTS
ci-runner revoked 2026-06-16T10:34:17.833Z -swamp worker connect
Connect this machine to an orchestrator as a remote execution worker.
swamp worker connect [url] --token <token> [--label k=v ...] [--cache-dir <dir>] [--data-plane-url <url>] [--concurrency <n>] [--no-reconnect]Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--token <token> |
Yes | Enrollment token in <name>.<secret> format |
-- |
--server-token <token> |
No | Server access token for authenticating the WebSocket connection (<name>.<secret>) |
-- |
--label <label> |
No | Scheduling label key=value (repeatable) |
-- |
--data-plane-url <url> |
No | Override the data-plane base URL | Connect URL over HTTP |
--cache-dir <dir> |
No | Bundle/asset cache directory; also stores the machine-id file the token binds to | Fresh temp dir (new identity per process) |
--max-dispatches <n> |
No | Drain and exit 0 after N dispatches complete (env: SWAMP_WORKER_MAX_DISPATCHES) |
-- |
--idle-timeout <dur> |
No | Drain and exit 0 after being continuously idle for this duration (env: SWAMP_WORKER_IDLE_TIMEOUT) |
-- |
--no-reconnect |
No | Exit when the control socket closes instead of reconnecting | Reconnect on disconnect |
Cache directory
Set --cache-dir to a stable directory so the worker can re-enroll after a
restart. Without it, the worker receives a fresh temp directory (and therefore a
new machine identity) on every process start. A stable cache directory preserves
the machine-id file that the enrollment token binds to, allowing the worker to
resume its identity across restarts.
Environment variables
| Variable | Flag | Description |
|---|---|---|
SWAMP_ORCHESTRATOR_URL |
<url> positional |
Orchestrator WebSocket URL (ws:// or wss://) |
SWAMP_WORKER_TOKEN |
--token |
Enrollment token in <name>.<secret> format |
SWAMP_SERVER_TOKEN |
--server-token |
Server access token in <name>.<secret> format |
SWAMP_WORKER_LABELS |
--label |
Comma-separated key=value pairs (e.g. region=us-east,gpu=true) |
SWAMP_WORKER_CACHE_DIR |
--cache-dir |
Bundle/asset cache directory |
SWAMP_WORKER_MAX_DISPATCHES |
--max-dispatches |
Drain and exit 0 after N dispatches complete |
SWAMP_WORKER_IDLE_TIMEOUT |
--idle-timeout |
Drain and exit 0 after being continuously idle for this duration |
SWAMP_WORKER_CONCURRENCY |
--concurrency |
Number of concurrent dispatch slots (auto = CPU count, min 1) |
Explicit flags take precedence when both a flag and its environment variable are set.
Examples
$ swamp worker connect wss://orchestrator.example.com \
--token ci-runner.9ce100bf00607d0edc3f231d6f7d85d028ba820b238fda113469cf0fdf7bfa97 \
--label env=ci --label arch=arm64 \
--cache-dir /var/lib/swamp-worker# Container/cloud-init: connect using only environment variables
SWAMP_ORCHESTRATOR_URL=wss://orch:9090 \
SWAMP_WORKER_TOKEN=ci-pool.a3f9... \
SWAMP_WORKER_LABELS=tier=ci,region=us-east \
SWAMP_WORKER_CACHE_DIR=/var/lib/swamp-worker \
swamp worker connect# Ephemeral worker: one dispatch, then exit
swamp worker connect wss://orch:9090 --token <token> --max-dispatches 1# Exit after 10 idle minutes
swamp worker connect wss://orch:9090 --token <token> --idle-timeout 10m# Big node: 8 concurrent dispatch slots
swamp worker connect wss://orch:9090 --token <token> --concurrency 8# Auto-detect from CPU count
swamp worker connect wss://orch:9090 --token <token> --concurrency auto# Connect to a token-authenticated orchestrator
swamp worker connect wss://orch:9090 \
--server-token admin.secret \
--token worker-pool.enrollment-secretServer token vs enrollment token
--token is the enrollment token — it identifies the worker to the
orchestrator and binds it to a machine identity. --server-token is the
server access token — it authenticates the WebSocket connection itself. A
server token is required when the orchestrator runs with
--auth-mode token. Without it,
the connection receives HTTP 401.
Use swamp auth server-login to store a server token for reuse, or set the
SWAMP_SERVER_TOKEN environment variable.
swamp worker list
List workers in the pool. By default, only connected and draining workers are
shown. Disconnected workers are hidden unless --all is passed.
swamp worker list [--all] [--server <url>] [--token <token>]Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--all |
No | Include disconnected workers | Hidden |
--server <url> |
No | Server URL | -- |
--token <token> |
No | Server token (--server) |
-- |
The STATUS column reports connected, disconnected, or draining. A
draining worker is finishing in-flight work before disconnecting and is
excluded from new dispatch scheduling. Without --all, only connected and
draining workers appear.
The SLOTS column shows the worker's current dispatch load as active/capacity
(e.g. 0/4, 3/8). Workers with --concurrency 1 (the default) show 0/1 or
1/1.
Table output
$ swamp worker listNAME STATUS LABELS PLATFORM SLOTS LAST SEEN
ci-runner-a1b2 connected env=ci linux/amd64 0/1 2026-07-04T10:34:30.000Z
gpu-fleet-c3d4 connected gpu=true linux/arm64 3/8 2026-07-04T10:34:31.000ZTable output with --all
$ swamp worker list --allNAME STATUS LABELS PLATFORM SLOTS LAST SEEN
ci-runner-a1b2 connected env=ci linux/amd64 0/1 2026-07-04T10:34:30.000Z
gpu-fleet-c3d4 connected gpu=true linux/arm64 3/8 2026-07-04T10:34:31.000Z
build-node-e5f6 disconnected tier=build linux/amd64 0/4 2026-07-03T22:15:00.000ZJSON output
$ swamp worker list --json{
"workers": [
{
"name": "ci-runner-a1b2",
"status": "connected",
"labels": { "env": "ci" },
"platform": "linux",
"arch": "amd64",
"capacity": 1,
"activeDispatchIds": [],
"lastSeen": "2026-07-04T10:34:30.000Z"
}
],
"count": 1,
"filteredDisconnectedCount": 1
}filteredDisconnectedCount is the number of disconnected workers omitted from
the workers array. With --all, this field is 0 and all workers are
included.
JSON fields (envelope)
| Field | Type | Description |
|---|---|---|
workers |
array |
List of worker objects |
count |
number |
Number of workers in the workers array |
filteredDisconnectedCount |
number |
Disconnected workers omitted (0 when --all is used) |
JSON fields (per worker)
| Field | Type | Description |
|---|---|---|
name |
string |
Worker name |
status |
string |
connected, disconnected, or draining |
labels |
object |
Key-value labels assigned at enrollment |
platform |
string |
Operating system |
arch |
string |
CPU architecture |
capacity |
number |
Total dispatch slots |
activeDispatchIds |
array |
IDs of dispatches currently running on this worker |
lastSeen |
string |
ISO 8601 timestamp of last heartbeat |
Empty pool
When the pool is empty:
$ swamp worker listNo workers found.
Workers appear here after enrolling with: swamp worker token create <name> --duration 24hswamp worker verify
Run a fleet probe against connected workers to confirm dispatch, capability RPC, and data-plane connectivity.
swamp worker verify [name] [--label k=v] [--server <url>]Arguments
| Argument | Required | Description |
|---|---|---|
name |
No | Verify a single worker by name. Omit to verify all. |
Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--label <label> |
No | Verify workers matching the label selector | -- |
--server <url> |
No | Server URL | -- |
Exits non-zero when any worker fails verification.
Table output
$ swamp worker verifyWORKER STATUS PLATFORM ARCH DETAILS
ci-runner-a1b2 pass linux amd64 -
gpu-fleet-c3d4 fail linux arm64 dataPlaneOk: falseJSON output
$ swamp worker verify --json{
"workers": [
{
"name": "ci-runner-a1b2",
"status": "pass",
"probeMarkerOk": true,
"queryOk": true,
"dataPlaneOk": true,
"failures": []
},
{
"name": "gpu-fleet-c3d4",
"status": "fail",
"probeMarkerOk": true,
"queryOk": true,
"dataPlaneOk": false,
"failures": ["data plane write+read round-trip failed"]
}
],
"total": 2,
"passed": 1,
"failed": 1
}JSON fields
| Field | Type | Description |
|---|---|---|
name |
string |
Worker name |
status |
string |
pass or fail |
probeMarkerOk |
boolean |
Dispatch-level metadata arrived intact |
queryOk |
boolean |
Capability RPC channel (worker to orchestrator) responded |
dataPlaneOk |
boolean |
HTTP data-plane write and read round-trip succeeded |
failures |
array |
List of failure detail strings (empty on pass) |
total |
number |
Total workers probed |
passed |
number |
Workers that passed all probes |
failed |
number |
Workers that failed one or more probes |
swamp worker queue
List steps currently queued for dispatch, waiting for a matching worker.
swamp worker queue [--server <url>] [--repo-dir <dir>]When steps are waiting:
$ swamp worker queueREQUIREMENT STEP MODEL QUEUED AT AGE
labels gpu=true train-model ml/trainer 2026-07-04T10:32:15.000Z 3m
target build-node compile builder/go 2026-07-04T10:34:02.000Z 1mWhen the queue is empty:
$ swamp worker queue
No steps are currently queued.JSON output
$ swamp worker queue --json[
{
"queueId": "a1b2c3d4-...",
"requirement": "labels gpu=true",
"stepName": "train-model",
"modelType": "ml/trainer",
"methodName": "run",
"workflowName": "training-pipeline",
"queuedAt": "2026-07-04T10:32:15.000Z",
"ageMs": 180000
}
]JSON fields
| Field | Type | Description |
|---|---|---|
queueId |
string |
Unique identifier for this queue episode |
requirement |
string |
Human-readable placement requirement |
stepName |
string |
Workflow step name (if available) |
modelType |
string |
Model type being dispatched |
methodName |
string |
Method being invoked |
workflowName |
string |
Workflow containing the step (if available) |
queuedAt |
string |
ISO 8601 timestamp when the step entered the queue |
ageMs |
number |
Milliseconds since the step was queued |
Steps appear here when they are waiting for a matching worker and disappear once dispatched, timed out, or cancelled. Use this to diagnose why a workflow is stalled and to inform autoscaling decisions.
The same data is queryable via swamp data query:
swamp data query 'modelType == "swamp/pending-dispatch" && attributes.state == "waiting"'swamp worker daemon enable
Generate and install a systemd unit (Linux) or launchd plist (macOS) that runs
swamp worker connect as a system service.
swamp worker daemon enable [url] --token <token> [--label k=v ...] [--cache-dir <dir>] [--max-dispatches <n>] [--idle-timeout <dur>]Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--token <token> |
Yes | Enrollment token in <name>.<secret> format |
-- |
--server-token <token> |
No | Server access token for authenticating the WebSocket connection (<name>.<secret>) |
-- |
--label <label> |
No | Scheduling label key=value (repeatable) |
-- |
--data-plane-url <url> |
No | Override the data-plane base URL | Connect URL over HTTP |
--cache-dir <dir> |
No | Bundle/asset cache directory; also stores the machine-id file the token binds to | /var/lib/swamp-worker (recommended) |
--max-dispatches <n> |
No | Drain and exit 0 after N dispatches complete (env: SWAMP_WORKER_MAX_DISPATCHES) |
-- |
--idle-timeout <dur> |
No | Drain and exit 0 after being continuously idle for this duration (env: SWAMP_WORKER_IDLE_TIMEOUT) |
-- |
--no-reconnect |
No | Exit when the control socket closes instead of reconnecting | Reconnect on disconnect |
Service settings
The generated unit file configures the following behavior:
| Setting | Description |
|---|---|
KillSignal=SIGTERM |
Triggers graceful drain — the worker finishes in-flight work before stopping |
TimeoutStopSec=300 |
5-minute window for the drain to complete before the service manager force-kills the process |
Restart=on-failure (systemd) / KeepAlive: {SuccessfulExit: false} (launchd) |
Restarts on crashes but not on policy-complete exits (drain, max-dispatches, idle-timeout) |
Token and other secrets are placed in environment directives inside the unit
file, which is installed with mode 0600. They do not appear in ExecStart
(systemd) or ProgramArguments (launchd).
--cache-dir should point to a stable path (e.g. /var/lib/swamp-worker) for
daemon mode so the machine identity survives service restarts.
Example
swamp worker daemon enable wss://orch:9090 \
--token pool.secret \
--label tier=ci --label region=us-east \
--cache-dir /var/lib/swamp-worker \
--idle-timeout 30m# Token-authenticated orchestrator
swamp worker daemon enable wss://orch:9090 \
--token pool.secret \
--server-token admin.secret \
--label tier=ciswamp worker daemon disable
Stop the service and remove the unit file.
swamp worker daemon disableswamp worker daemon status
Show whether the daemon is installed, running, its PID, and log location.
swamp worker daemon status [--json]Table output
$ swamp worker daemon statusDaemon: installed
State: running
PID: 48201
Log: /var/log/swamp-worker.logJSON output
$ swamp worker daemon status --json{
"installed": true,
"running": true,
"pid": 48201,
"logPath": "/var/log/swamp-worker.log"
}swamp serve
Start a WebSocket API server for workflow and model execution. Workers connect to this server.
swamp serve [--port <port>] [--host <host>] [--cert-file <path>] [--key-file <path>] [--no-schedule] [--webhook <spec>] [--repo-dir <dir>]Flags
| Flag | Required | Description | Default |
|---|---|---|---|
--port <port> |
No | Port to listen on | 9090 |
--host <host> |
No | Host to bind to | 127.0.0.1 |
--cert-file <path> |
No | PEM-encoded certificate file for TLS | (unset) |
--key-file <path> |
No | PEM-encoded private key file for TLS | (unset) |
--no-schedule |
No | Disable scheduled workflow execution | Schedules enabled |
--webhook <spec> |
No | Register a webhook endpoint: <route>:<workflow>:<secret> |
-- |
--repo-dir <dir> |
No | Repository directory | . |
When both --cert-file and --key-file are provided, the server listens over
wss:///https:// instead of ws:///http://. See
TLS and Proxies for details.
Service deployments
Swamp loads all extensions through an embedded runtime under ~/.swamp. When
running under a service manager (e.g. systemd), ensure HOME is set in the unit
environment so the runtime can locate its extension store.
Example
$ swamp serve --port 8080 --host 0.0.0.0 --webhook /deploy:ci-deploy:s3cr3tExit-code contract
swamp worker connect uses its exit code to distinguish intentional shutdown
from failure.
| Exit code | Meaning |
|---|---|
| 0 | Policy-complete: drained via signal, --max-dispatches reached, or --idle-timeout fired |
| Non-zero | Failure: permanent enrollment rejection, unexpected error |
See Failure Semantics — Graceful drain for the full drain sequence.
Related
- Enrollment Tokens — token lifecycle, machine binding, expiry
- Workflow Placement — step placement fields and scheduling
- Failure Semantics — reconnection, in-flight dispatch, cancellation
- Remote Execution — overview and core concepts
- TLS and Proxies — trust stores and
server-side TLS for
swamp serve - Serve Flags — all
swamp serveflags including authentication and access control