Skip to main content

REMOTE EXECUTION

This guide covers day-to-day operations for a running remote execution setup: monitoring the worker pool, deploying workers to different environments, diagnosing enrollment failures, and upgrading the fleet.

For the initial setup walkthrough, see the Remote Execution tutorial. For full CLI flag and field reference, see Worker Commands.

Monitor the worker pool

To see connected workers with their status, labels, platform, and last-seen time, run:

swamp worker list

To see all enrollment tokens with their state and expiry:

swamp worker token list

For deeper inspection, query the built-in behaviors with swamp data:

# All workers and their current state
swamp data query 'modelType == "swamp/worker"'

# All enrollment tokens, including expired and revoked
swamp data query 'modelType == "swamp/enrollment-token"'

# Active step leases (steps currently dispatched to workers)
swamp data query 'modelType == "swamp/step-lease"'

These built-in behaviors expose the orchestrator's internal state as queryable data, so you can filter, project, and pipe the output through --json the same way you would with any other Swamp data query. See Data for the full query syntax.

Deploy a worker

Each deployment scenario below connects a worker to an orchestrator using swamp worker connect. Adjust the orchestrator URL, token, and --cache-dir path for your environment. For the full flag reference, see Worker Commands.

SSH

ssh worker-host 'swamp worker connect wss://orchestrator:9090 --token <token> --cache-dir /var/lib/swamp-worker'

Container

The container and Kubernetes examples use the swamp-worker image from Run a Worker in Docker. Its entrypoint runs swamp, so pass only the subcommand and flags:

docker run -d swamp-worker \
    worker connect wss://orchestrator:9090 \
    --token <token> \
    --cache-dir /data

Kubernetes Job

apiVersion: batch/v1
kind: Job
metadata:
  name: swamp-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: swamp-worker:latest
          args:
            - worker
            - connect
            - wss://orchestrator:9090
            - --token
            - $(WORKER_TOKEN)
            - --cache-dir
            - /data
          env:
            - name: WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
          volumeMounts:
            - name: cache
              mountPath: /data
      volumes:
        - name: cache
          emptyDir: {}
      restartPolicy: OnFailure

CI runner step

# GitHub Actions example
- name: Connect Swamp worker
  run: |
    swamp worker connect wss://orchestrator:9090 \
      --token ${{ secrets.SWAMP_WORKER_TOKEN }} \
      --cache-dir /tmp/swamp-worker

Cloud-init / user-data

#cloud-config
runcmd:
  - curl -fsSL https://swamp-club.com/install.sh | sh
  - swamp worker connect wss://orchestrator:9090
    --token <token>
    --cache-dir /var/lib/swamp-worker

In all cases, set --cache-dir to a stable path so the worker preserves its machine identity across restarts. Without it, the worker generates a new identity on each start and the orchestrator treats it as a different machine. See Enrollment Tokens for how token binding works.

Troubleshoot enrollment failures

When swamp worker connect fails, the error message identifies the cause. The table below covers all permanent enrollment failure messages:

Error message Cause Fix
protocol_mismatch Worker and orchestrator are on different Swamp versions Upgrade both to the same version
already_connected A worker with this token is already connected Check for duplicate worker processes; revoke and re-create the token
already bound to another machine Token was enrolled by a different machine identity Use a new token, or restore the original --cache-dir
expired Token lifetime has elapsed Create a new token with swamp worker token create
revoked Token was manually revoked Create a new token
does not match Token secret doesn't match the stored hash Check for typos in the token value; create a new token if needed

For the full token state machine and lifecycle details, see Enrollment Tokens.

Clean up stale workers

In ephemeral fleet deployments — CI runners, spot instances, autoscaled containers — workers disconnect when their host is terminated. These disconnected entries accumulate in the pool over time.

Use swamp worker prune to remove workers that have been disconnected longer than a grace period:

# Preview what would be removed
swamp worker prune --dry-run

# Remove workers disconnected for more than 7 days
swamp worker prune --grace-period 7d --force

# Prune against a remote server
swamp worker prune --grace-period 24h --server wss://orch:9090

The default grace period is 24 hours. Connected and draining workers are never pruned. swamp serve also removes disconnected workers on its own, hourly, after 24 hours, or after garbageCollection.bookkeeping when automatic is on. For the full flag reference, see Worker Commands — swamp worker prune.

Upgrade the fleet

The orchestrator and workers must run the same Swamp version. The WebSocket protocol does not negotiate between versions — a version mismatch produces a protocol_mismatch error and the worker disconnects.

To upgrade:

  1. Upgrade the orchestrator first, then upgrade all workers. Alternatively, upgrade all machines at once.
  2. Workers running an older version will see protocol_mismatch and disconnect until they are upgraded.
  3. Rolling upgrades are not supported. All workers must be on the same version as the orchestrator before they can reconnect.
← PreviousInspect Repo ActivityNext →Worker Fleets