Skip to main content

RUN A FLEET ON KUBERNETES

This guide shows you how to deploy a Swamp worker fleet to Kubernetes using minikube for local testing. The same manifests work on any Kubernetes cluster with adjusted image loading and orchestrator URLs.

Prerequisites

You need a running orchestrator with TLS and token authentication on the host machine, on Swamp v20261008.183339.0-sha.231eda34 or later and started while swamp-club.com is reachable. See Run a Worker in Docker for the setup steps. When generating the TLS certificate, include host.minikube.internal in the SAN and add it to --trusted-hosts:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
  -pkeyopt ec_param_enc:named_curve \
  -keyout key.pem -out cert.pem -days 30 -nodes \
  -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,DNS:host.docker.internal,DNS:host.minikube.internal" \
  -addext "basicConstraints=CA:FALSE"

swamp serve --host 0.0.0.0 \
  --cert-file cert.pem --key-file key.pem \
  --auth-mode token --admins 'user:<you>' \
  --trusted-hosts host.docker.internal,host.minikube.internal

Load the image into minikube

docker build -t swamp-worker .
minikube image load swamp-worker

This uses the same Dockerfile from Run a Worker in Docker, which installs Swamp via the install script.

Create secrets

Create a fleet token and store it alongside the server token and TLS CA certificate as Kubernetes Secrets. The server token is the worker-token value from Create tokens, printed by swamp access token reveal worker-token --yes:

swamp worker token create k8s-fleet --duration 7d --max-enrollments 50

kubectl create secret generic swamp-worker-token \
  --from-literal=token='k8s-fleet.<secret>'

kubectl create secret generic swamp-server-token \
  --from-literal=token='worker-token.<secret>'

kubectl create secret generic swamp-tls-ca \
  --from-file=ca.pem=cert.pem

The worker pods need no swamp-club.com key: they pass the account check through the orchestrator that enrolls them. See Remote workers.

Find the orchestrator URL

The orchestrator runs on the host machine. The URL depends on your platform:

  • macOS / Windows: wss://host.minikube.internal:9090

  • Linux: use the gateway IP from the minikube network:

    minikube ssh -- route -n | awk '/^0.0.0.0/{print $2}'

    Then use wss://<gateway-ip>:9090.

Deploy as a Deployment

A Deployment keeps a fixed number of workers running. Use this for a persistent pool that stays available for incoming work.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: swamp-workers
spec:
  replicas: 3
  selector:
    matchLabels:
      app: swamp-worker
  template:
    metadata:
      labels:
        app: swamp-worker
    spec:
      terminationGracePeriodSeconds: 300
      containers:
        - name: worker
          image: swamp-worker
          imagePullPolicy: Never
          args: ["worker", "connect"]
          env:
            - name: SWAMP_ORCHESTRATOR_URL
              value: wss://host.minikube.internal:9090
            - name: SWAMP_SERVER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-server-token
                  key: token
            - name: SWAMP_WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
            - name: SWAMP_WORKER_CONCURRENCY
              value: "2"
            - name: SWAMP_CA_CERT
              value: /certs/ca.pem
          volumeMounts:
            - name: tls-ca
              mountPath: /certs
              readOnly: true
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
      volumes:
        - name: tls-ca
          secret:
            secretName: swamp-tls-ca

The manifest sets args rather than command, because command replaces the image's tini entrypoint and leaves orphaned step processes unreaped.

terminationGracePeriodSeconds: 300 gives in-flight steps five minutes to complete during pod termination. Kubernetes sends SIGTERM first; the worker drains its dispatches before exiting.

The memory request is 1Gi because each worker spawns a child process for dispatch execution. Under-provisioned workers are killed by the OOM killer mid-step.

If you are on Linux, replace host.minikube.internal with the gateway IP from the step above.

Apply and verify:

kubectl apply -f deployment.yaml
swamp worker list

Scale the Deployment

kubectl scale deployment swamp-workers --replicas=5

To scale down, Kubernetes terminates excess pods. Each pod drains its in-flight work during the grace period before shutting down.

kubectl scale deployment swamp-workers --replicas=1

Deploy as a Job

A Job runs a worker that processes a fixed number of dispatches and exits. Use this for batch processing where each worker handles one step and is replaced.

apiVersion: batch/v1
kind: Job
metadata:
  name: swamp-batch-worker
spec:
  completions: 5
  parallelism: 5
  template:
    spec:
      terminationGracePeriodSeconds: 300
      containers:
        - name: worker
          image: swamp-worker
          imagePullPolicy: Never
          args: ["worker", "connect"]
          env:
            - name: SWAMP_ORCHESTRATOR_URL
              value: wss://host.minikube.internal:9090
            - name: SWAMP_SERVER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-server-token
                  key: token
            - name: SWAMP_WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
            - name: SWAMP_WORKER_MAX_DISPATCHES
              value: "1"
            - name: SWAMP_CA_CERT
              value: /certs/ca.pem
          volumeMounts:
            - name: tls-ca
              mountPath: /certs
              readOnly: true
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
      volumes:
        - name: tls-ca
          secret:
            secretName: swamp-tls-ca
      restartPolicy: Never

SWAMP_WORKER_MAX_DISPATCHES=1 tells the worker to drain and exit after one dispatch. Combined with restartPolicy: Never, each pod processes one step and terminates.

← PreviousRun a Fleet with Docker ComposeNext →Scale from Zero Workers