Skip to main content

WORKER COMMANDS

CLI reference for managing remote execution workers and the Swamp server. This page covers token lifecycle commands, worker enrollment, pool inspection, queue visibility, and the swamp serve entry point.

For enrollment token lifecycle details (machine binding, expiry, state transitions), see Enrollment Tokens. For step-level placement fields that route work to specific workers, see Workflow Placement.

Global flags

All commands in this section accept the standard global options. These are documented once here rather than repeated per command.

Flag Description
--json Emit structured JSON instead of human-readable output
--log <path> Write log output to the given file
--log-level <lvl> Set log verbosity (debug, info, warn, error)
-q Quiet mode — suppress non-essential output
-v Verbose mode — increase output detail
--no-telemetry Disable telemetry for this invocation
--show-properties Print resolved configuration properties
--no-color Disable colored output

swamp worker token create

Mint a named worker enrollment token. The plaintext is shown exactly once in <name>.<secret> format.

swamp worker token create <name> --duration <duration> [--vault <vault>] [--max-enrollments <n>]

Flags

Flag Required Description Default
--duration <duration> Yes Token lifetime (e.g. 30m, 1h, 24h, 7d). Hard deadline: the enrolled worker is disconnected when it elapses. --
--vault <vault> No Vault that stores the token plaintext Sole configured vault
--max-enrollments <n> No Maximum machines this token can enroll (positive integer or "unlimited"). Fleet tokens name workers as <token>-<suffix> and auto-inject a fleet=<token> label. 1

Example

$ swamp worker token create ci-runner --duration 24h
Token: ci-runner
Expires: 2026-06-16T10:34:17.833Z
Vault: sandbox-vault (key worker-token-ci-runner)

  ci-runner.9ce100bf00607d0edc3f231d6f7d85d028ba820b238fda113469cf0fdf7bfa97

This token is shown once and will not be displayed again — store it now.

The token value (ci-runner.9ce100bf...) is the credential passed to swamp worker connect --token. The name prefix before the dot identifies the token; the suffix after the dot is the secret. See Enrollment Tokens for the full token lifecycle.

Fleet token example

$ swamp worker token create gpu-fleet --duration 7d --max-enrollments 10
Token: gpu-fleet
Expires: 2026-07-11T00:00:00.000Z
Vault: sandbox-vault (key worker-token-gpu-fleet)
Max enrollments: 10

  gpu-fleet.a4e29c1f00817b3eac5f642d7e8c96b139ca931b349fea224580de1fef8cbaa8

This token is shown once and will not be displayed again — store it now.

swamp worker token list

List worker enrollment tokens showing recorded vs effective (display) state, expiry, and enrollments.

swamp worker token list

Table output

$ swamp worker token list
NAME        STATE     EXPIRES                   ENROLLMENTS
ci-runner   enrolled  2026-06-16T10:34:17.833Z  machine-42
gpu-fleet   enrolled  2026-07-11T00:00:00.000Z  3 / 10

Single-enrollment tokens show the bound machine ID (e.g. machine-42). Fleet tokens show used / allowance (e.g. 3 / 10, 8 / unlimited).

JSON output

$ swamp worker token list --json
[
  {
    "name": "ci-runner",
    "state": "enrolled",
    "effectiveState": "enrolled",
    "createdAt": "2026-06-15T10:34:17.833Z",
    "expiresAt": "2026-06-16T10:34:17.833Z",
    "maxEnrollments": 1,
    "bindingCount": 1,
    "bindings": [
      { "machineId": "machine-42", "enrolledAt": "2026-06-15T10:34:30.000Z" }
    ],
    "vaultName": "sandbox-vault",
    "secretKey": "worker-token-ci-runner"
  }
]

JSON fields

Field Type Description
name string Token name as provided at creation
state string Recorded state (unused, enrolled, revoked)
effectiveState string Display state accounting for expiry (may differ from state)
createdAt string ISO 8601 creation timestamp
expiresAt string ISO 8601 expiry timestamp
maxEnrollments number Maximum machines this token can enroll (1 for single-enrollment tokens)
bindingCount number Number of machines currently bound to this token
bindings array List of bound machines, each with machineId (string) and enrolledAt (ISO 8601)
vaultName string Vault storing the token plaintext
secretKey string Key under which the token is stored in the vault

swamp worker token revoke

Invalidate a worker enrollment token before it expires.

swamp worker token revoke <name>

Example

$ swamp worker token revoke ci-runner
✓ Token ci-runner revoked.
Revoked at: 2026-06-15T10:34:39.736Z

After revocation, swamp worker token list reflects the updated state:

NAME       STATE    EXPIRES                   ENROLLMENTS
ci-runner  revoked  2026-06-16T10:34:17.833Z  -

swamp worker connect

Connect this machine to an orchestrator as a remote execution worker.

swamp worker connect [url] --token <token> [--label k=v ...] [--cache-dir <dir>] [--data-plane-url <url>] [--concurrency <n>] [--no-reconnect]

Flags

Flag Required Description Default
--token <token> Yes Enrollment token in <name>.<secret> format --
--server-token <token> No Server access token for authenticating the WebSocket connection (<name>.<secret>) --
--label <label> No Scheduling label key=value (repeatable) --
--data-plane-url <url> No Override the data-plane base URL Connect URL over HTTP
--cache-dir <dir> No Bundle/asset cache directory; also stores the machine-id file the token binds to Fresh temp dir (new identity per process)
--max-dispatches <n> No Drain and exit 0 after N dispatches complete (env: SWAMP_WORKER_MAX_DISPATCHES) --
--idle-timeout <dur> No Drain and exit 0 after being continuously idle for this duration (env: SWAMP_WORKER_IDLE_TIMEOUT) --
--no-reconnect No Exit when the control socket closes instead of reconnecting Reconnect on disconnect

Cache directory

Set --cache-dir to a stable directory so the worker can re-enroll after a restart. Without it, the worker receives a fresh temp directory (and therefore a new machine identity) on every process start. A stable cache directory preserves the machine-id file that the enrollment token binds to, allowing the worker to resume its identity across restarts.

Environment variables

Variable Flag Description
SWAMP_ORCHESTRATOR_URL <url> positional Orchestrator WebSocket URL (ws:// or wss://)
SWAMP_WORKER_TOKEN --token Enrollment token in <name>.<secret> format
SWAMP_SERVER_TOKEN --server-token Server access token in <name>.<secret> format
SWAMP_WORKER_LABELS --label Comma-separated key=value pairs (e.g. region=us-east,gpu=true)
SWAMP_WORKER_CACHE_DIR --cache-dir Bundle/asset cache directory
SWAMP_WORKER_MAX_DISPATCHES --max-dispatches Drain and exit 0 after N dispatches complete
SWAMP_WORKER_IDLE_TIMEOUT --idle-timeout Drain and exit 0 after being continuously idle for this duration
SWAMP_WORKER_CONCURRENCY --concurrency Number of concurrent dispatch slots (auto = CPU count, min 1)

Explicit flags take precedence when both a flag and its environment variable are set.

Examples

$ swamp worker connect wss://orchestrator.example.com \
    --token ci-runner.9ce100bf00607d0edc3f231d6f7d85d028ba820b238fda113469cf0fdf7bfa97 \
    --label env=ci --label arch=arm64 \
    --cache-dir /var/lib/swamp-worker
# Container/cloud-init: connect using only environment variables
SWAMP_ORCHESTRATOR_URL=wss://orch:9090 \
SWAMP_WORKER_TOKEN=ci-pool.a3f9... \
SWAMP_WORKER_LABELS=tier=ci,region=us-east \
SWAMP_WORKER_CACHE_DIR=/var/lib/swamp-worker \
swamp worker connect
# Ephemeral worker: one dispatch, then exit
swamp worker connect wss://orch:9090 --token <token> --max-dispatches 1
# Exit after 10 idle minutes
swamp worker connect wss://orch:9090 --token <token> --idle-timeout 10m
# Big node: 8 concurrent dispatch slots
swamp worker connect wss://orch:9090 --token <token> --concurrency 8
# Auto-detect from CPU count
swamp worker connect wss://orch:9090 --token <token> --concurrency auto
# Connect to a token-authenticated orchestrator
swamp worker connect wss://orch:9090 \
    --server-token admin.secret \
    --token worker-pool.enrollment-secret

Server token vs enrollment token

--token is the enrollment token — it identifies the worker to the orchestrator and binds it to a machine identity. --server-token is the server access token — it authenticates the WebSocket connection itself. A server token is required when the orchestrator runs with --auth-mode token. Without it, the connection receives HTTP 401.

Use swamp auth server-login to store a server token for reuse, or set the SWAMP_SERVER_TOKEN environment variable.


swamp worker list

List workers in the pool. By default, only connected and draining workers are shown. Disconnected workers are hidden unless --all is passed.

swamp worker list [--all] [--server <url>] [--token <token>]

Flags

Flag Required Description Default
--all No Include disconnected workers Hidden
--server <url> No Server URL --
--token <token> No Server token (--server) --

The STATUS column reports connected, disconnected, or draining. A draining worker is finishing in-flight work before disconnecting and is excluded from new dispatch scheduling. Without --all, only connected and draining workers appear.

The SLOTS column shows the worker's current dispatch load as active/capacity (e.g. 0/4, 3/8). Workers with --concurrency 1 (the default) show 0/1 or 1/1.

Table output

$ swamp worker list
NAME            STATUS     LABELS          PLATFORM      SLOTS  LAST SEEN
ci-runner-a1b2  connected  env=ci          linux/amd64   0/1    2026-07-04T10:34:30.000Z
gpu-fleet-c3d4  connected  gpu=true        linux/arm64   3/8    2026-07-04T10:34:31.000Z

Table output with --all

$ swamp worker list --all
NAME            STATUS        LABELS          PLATFORM      SLOTS  LAST SEEN
ci-runner-a1b2  connected     env=ci          linux/amd64   0/1    2026-07-04T10:34:30.000Z
gpu-fleet-c3d4  connected     gpu=true        linux/arm64   3/8    2026-07-04T10:34:31.000Z
build-node-e5f6 disconnected  tier=build      linux/amd64   0/4    2026-07-03T22:15:00.000Z

JSON output

$ swamp worker list --json
{
  "workers": [
    {
      "name": "ci-runner-a1b2",
      "status": "connected",
      "labels": { "env": "ci" },
      "platform": "linux",
      "arch": "amd64",
      "capacity": 1,
      "activeDispatchIds": [],
      "lastSeen": "2026-07-04T10:34:30.000Z"
    }
  ],
  "count": 1,
  "filteredDisconnectedCount": 1
}

filteredDisconnectedCount is the number of disconnected workers omitted from the workers array. With --all, this field is 0 and all workers are included.

JSON fields (envelope)

Field Type Description
workers array List of worker objects
count number Number of workers in the workers array
filteredDisconnectedCount number Disconnected workers omitted (0 when --all is used)

JSON fields (per worker)

Field Type Description
name string Worker name
status string connected, disconnected, or draining
labels object Key-value labels assigned at enrollment
platform string Operating system
arch string CPU architecture
capacity number Total dispatch slots
activeDispatchIds array IDs of dispatches currently running on this worker
lastSeen string ISO 8601 timestamp of last heartbeat

Empty pool

When the pool is empty:

$ swamp worker list
No workers found.
Workers appear here after enrolling with: swamp worker token create <name> --duration 24h

swamp worker verify

Run a fleet probe against connected workers to confirm dispatch, capability RPC, and data-plane connectivity.

swamp worker verify [name] [--label k=v] [--server <url>]

Arguments

Argument Required Description
name No Verify a single worker by name. Omit to verify all.

Flags

Flag Required Description Default
--label <label> No Verify workers matching the label selector --
--server <url> No Server URL --

Exits non-zero when any worker fails verification.

Table output

$ swamp worker verify
WORKER         STATUS  PLATFORM  ARCH    DETAILS
ci-runner-a1b2 pass    linux     amd64   -
gpu-fleet-c3d4 fail    linux     arm64   dataPlaneOk: false

JSON output

$ swamp worker verify --json
{
  "workers": [
    {
      "name": "ci-runner-a1b2",
      "status": "pass",
      "probeMarkerOk": true,
      "queryOk": true,
      "dataPlaneOk": true,
      "failures": []
    },
    {
      "name": "gpu-fleet-c3d4",
      "status": "fail",
      "probeMarkerOk": true,
      "queryOk": true,
      "dataPlaneOk": false,
      "failures": ["data plane write+read round-trip failed"]
    }
  ],
  "total": 2,
  "passed": 1,
  "failed": 1
}

JSON fields

Field Type Description
name string Worker name
status string pass or fail
probeMarkerOk boolean Dispatch-level metadata arrived intact
queryOk boolean Capability RPC channel (worker to orchestrator) responded
dataPlaneOk boolean HTTP data-plane write and read round-trip succeeded
failures array List of failure detail strings (empty on pass)
total number Total workers probed
passed number Workers that passed all probes
failed number Workers that failed one or more probes

swamp worker queue

List steps currently queued for dispatch, waiting for a matching worker.

swamp worker queue [--server <url>] [--repo-dir <dir>]

When steps are waiting:

$ swamp worker queue
REQUIREMENT          STEP              MODEL          QUEUED AT                   AGE
labels gpu=true      train-model       ml/trainer     2026-07-04T10:32:15.000Z    3m
target build-node    compile           builder/go     2026-07-04T10:34:02.000Z    1m

When the queue is empty:

$ swamp worker queue
No steps are currently queued.

JSON output

$ swamp worker queue --json
[
  {
    "queueId": "a1b2c3d4-...",
    "requirement": "labels gpu=true",
    "stepName": "train-model",
    "modelType": "ml/trainer",
    "methodName": "run",
    "workflowName": "training-pipeline",
    "queuedAt": "2026-07-04T10:32:15.000Z",
    "ageMs": 180000
  }
]

JSON fields

Field Type Description
queueId string Unique identifier for this queue episode
requirement string Human-readable placement requirement
stepName string Workflow step name (if available)
modelType string Model type being dispatched
methodName string Method being invoked
workflowName string Workflow containing the step (if available)
queuedAt string ISO 8601 timestamp when the step entered the queue
ageMs number Milliseconds since the step was queued

Steps appear here when they are waiting for a matching worker and disappear once dispatched, timed out, or cancelled. Use this to diagnose why a workflow is stalled and to inform autoscaling decisions.

The same data is queryable via swamp data query:

swamp data query 'modelType == "swamp/pending-dispatch" && attributes.state == "waiting"'

swamp worker daemon enable

Generate and install a systemd unit (Linux) or launchd plist (macOS) that runs swamp worker connect as a system service.

swamp worker daemon enable [url] --token <token> [--label k=v ...] [--cache-dir <dir>] [--max-dispatches <n>] [--idle-timeout <dur>]

Flags

Flag Required Description Default
--token <token> Yes Enrollment token in <name>.<secret> format --
--server-token <token> No Server access token for authenticating the WebSocket connection (<name>.<secret>) --
--label <label> No Scheduling label key=value (repeatable) --
--data-plane-url <url> No Override the data-plane base URL Connect URL over HTTP
--cache-dir <dir> No Bundle/asset cache directory; also stores the machine-id file the token binds to /var/lib/swamp-worker (recommended)
--max-dispatches <n> No Drain and exit 0 after N dispatches complete (env: SWAMP_WORKER_MAX_DISPATCHES) --
--idle-timeout <dur> No Drain and exit 0 after being continuously idle for this duration (env: SWAMP_WORKER_IDLE_TIMEOUT) --
--no-reconnect No Exit when the control socket closes instead of reconnecting Reconnect on disconnect

Service settings

The generated unit file configures the following behavior:

Setting Description
KillSignal=SIGTERM Triggers graceful drain — the worker finishes in-flight work before stopping
TimeoutStopSec=300 5-minute window for the drain to complete before the service manager force-kills the process
Restart=on-failure (systemd) / KeepAlive: {SuccessfulExit: false} (launchd) Restarts on crashes but not on policy-complete exits (drain, max-dispatches, idle-timeout)

Token and other secrets are placed in environment directives inside the unit file, which is installed with mode 0600. They do not appear in ExecStart (systemd) or ProgramArguments (launchd).

--cache-dir should point to a stable path (e.g. /var/lib/swamp-worker) for daemon mode so the machine identity survives service restarts.

Example

swamp worker daemon enable wss://orch:9090 \
  --token pool.secret \
  --label tier=ci --label region=us-east \
  --cache-dir /var/lib/swamp-worker \
  --idle-timeout 30m
# Token-authenticated orchestrator
swamp worker daemon enable wss://orch:9090 \
  --token pool.secret \
  --server-token admin.secret \
  --label tier=ci

swamp worker daemon disable

Stop the service and remove the unit file.

swamp worker daemon disable

swamp worker daemon status

Show whether the daemon is installed, running, its PID, and log location.

swamp worker daemon status [--json]

Table output

$ swamp worker daemon status
Daemon:   installed
State:    running
PID:      48201
Log:      /var/log/swamp-worker.log

JSON output

$ swamp worker daemon status --json
{
  "installed": true,
  "running": true,
  "pid": 48201,
  "logPath": "/var/log/swamp-worker.log"
}

swamp serve

Start a WebSocket API server for workflow and model execution. Workers connect to this server.

swamp serve [--port <port>] [--host <host>] [--cert-file <path>] [--key-file <path>] [--no-schedule] [--webhook <spec>] [--repo-dir <dir>]

Flags

Flag Required Description Default
--port <port> No Port to listen on 9090
--host <host> No Host to bind to 127.0.0.1
--cert-file <path> No PEM-encoded certificate file for TLS (unset)
--key-file <path> No PEM-encoded private key file for TLS (unset)
--no-schedule No Disable scheduled workflow execution Schedules enabled
--webhook <spec> No Register a webhook endpoint: <route>:<workflow>:<secret> --
--repo-dir <dir> No Repository directory .

When both --cert-file and --key-file are provided, the server listens over wss:///https:// instead of ws:///http://. See TLS and Proxies for details.

Service deployments

Swamp loads all extensions through an embedded runtime under ~/.swamp. When running under a service manager (e.g. systemd), ensure HOME is set in the unit environment so the runtime can locate its extension store.

Example

$ swamp serve --port 8080 --host 0.0.0.0 --webhook /deploy:ci-deploy:s3cr3t

Exit-code contract

swamp worker connect uses its exit code to distinguish intentional shutdown from failure.

Exit code Meaning
0 Policy-complete: drained via signal, --max-dispatches reached, or --idle-timeout fired
Non-zero Failure: permanent enrollment rejection, unexpected error

See Failure Semantics — Graceful drain for the full drain sequence.