Remote Execution

This guide covers day-to-day operations for a running remote execution setup: monitoring the worker pool, deploying workers to different environments, diagnosing enrollment failures, and upgrading the fleet.

For the initial setup walkthrough, see the Remote Execution tutorial. For full CLI flag and field reference, see Worker Commands.

Monitor the worker pool

To see connected workers with their status, labels, platform, and last-seen time, run:

swamp worker list

To see all enrollment tokens with their state and expiry:

swamp worker token list

For deeper inspection, query the built-in models with swamp data:

# All workers and their current state
swamp data query 'modelType == "swamp/worker"'

# All enrollment tokens, including expired and revoked
swamp data query 'modelType == "swamp/enrollment-token"'

# Active step leases (steps currently dispatched to workers)
swamp data query 'modelType == "swamp/step-lease"'

These built-in models expose the orchestrator's internal state as queryable data, so you can filter, project, and pipe the output through --json the same way you would with any other Swamp data query. See Data Querying for the full query syntax.

Deploy a worker

Each deployment scenario below connects a worker to an orchestrator using swamp worker connect. Adjust the orchestrator URL, token, and --cache-dir path for your environment. For the full flag reference, see Worker Commands.

SSH

ssh worker-host 'swamp worker connect wss://orchestrator:9090 --token <token> --cache-dir /var/lib/swamp-worker'

Container

docker run -d swamp-worker \
    swamp worker connect wss://orchestrator:9090 \
    --token <token> \
    --cache-dir /data

Kubernetes Job

apiVersion: batch/v1
kind: Job
metadata:
  name: swamp-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: swamp-worker:latest
          command:
            - swamp
            - worker
            - connect
            - wss://orchestrator:9090
            - --token
            - $(WORKER_TOKEN)
            - --cache-dir
            - /data
          env:
            - name: WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
          volumeMounts:
            - name: cache
              mountPath: /data
      volumes:
        - name: cache
          emptyDir: {}
      restartPolicy: OnFailure

CI runner step

# GitHub Actions example
- name: Connect Swamp worker
  run: |
    swamp worker connect wss://orchestrator:9090 \
      --token ${{ secrets.SWAMP_WORKER_TOKEN }} \
      --cache-dir /tmp/swamp-worker

Cloud-init / user-data

#cloud-config
runcmd:
  - curl -fsSL https://swamp-club.com/install.sh | sh
  - swamp worker connect wss://orchestrator:9090
    --token <token>
    --cache-dir /var/lib/swamp-worker

In all cases, set --cache-dir to a stable path so the worker preserves its machine identity across restarts. Without it, the worker generates a new identity on each start and the orchestrator treats it as a different machine. See Enrollment Tokens for how token binding works.

Troubleshoot enrollment failures

When swamp worker connect fails, the error message identifies the cause. The table below covers all permanent enrollment failure messages:

Error message	Cause	Fix
`protocol_mismatch`	Worker and orchestrator are on different Swamp versions	Upgrade both to the same version
`already_connected`	A worker with this token is already connected	Check for duplicate worker processes; revoke and re-create the token
`already bound to another machine`	Token was enrolled by a different machine identity	Use a new token, or restore the original `--cache-dir`
`expired`	Token lifetime has elapsed	Create a new token with `swamp worker token create`
`revoked`	Token was manually revoked	Create a new token
`does not match`	Token secret doesn't match the stored hash	Check for typos in the token value; create a new token if needed

For the full token state machine and lifecycle details, see Enrollment Tokens.

Upgrade the fleet

The orchestrator and workers must run the same Swamp version. The WebSocket protocol does not negotiate between versions — a version mismatch produces a protocol_mismatch error and the worker disconnects.

To upgrade:

Upgrade the orchestrator first, then upgrade all workers. Alternatively, upgrade all machines at once.
Workers running an older version will see protocol_mismatch and disconnect until they are upgraded.
Rolling upgrades are not supported. All workers must be on the same version as the orchestrator before they can reconnect.

Remote Execution tutorial — getting-started walkthrough
Worker Commands — full CLI reference
Enrollment Tokens — token lifecycle details
Remote Execution — architecture and design rationale