Skip to main content

RUN A FLEET ON KUBERNETES

This guide shows you how to deploy a Swamp worker fleet to Kubernetes using minikube for local testing. The same manifests work on any Kubernetes cluster with adjusted image loading and orchestrator URLs.

Prerequisites

You need a running orchestrator with TLS and token authentication on the host machine. See Run a Worker in Docker for the setup steps. When generating the TLS certificate, include host.minikube.internal in the SAN and add it to --trusted-hosts:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
  -keyout key.pem -out cert.pem -days 30 -nodes \
  -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,DNS:host.docker.internal,DNS:host.minikube.internal" \
  -addext "basicConstraints=CA:FALSE"

swamp serve --host 0.0.0.0 \
  --cert-file cert.pem --key-file key.pem \
  --auth-mode token \
  --trusted-hosts host.docker.internal,host.minikube.internal

Load the image into minikube

swamp compile
docker build -t swamp-worker .
minikube image load swamp-worker

Create secrets

Create a fleet token and store it alongside the server token and TLS CA certificate as Kubernetes Secrets:

swamp worker token create k8s-fleet --duration 7d --max-enrollments 50

kubectl create secret generic swamp-worker-token \
  --from-literal=token='k8s-fleet.<secret>'

kubectl create secret generic swamp-server-token \
  --from-literal=token='admin.<secret>'

kubectl create secret generic swamp-tls-ca \
  --from-file=ca.pem=cert.pem

Find the orchestrator URL

The orchestrator runs on the host machine. The URL depends on your platform:

  • macOS / Windows: wss://host.minikube.internal:9090

  • Linux: use the gateway IP from the minikube network:

    minikube ssh -- route -n | awk '/^0.0.0.0/{print $2}'

    Then use wss://<gateway-ip>:9090.

Deploy as a Deployment

A Deployment keeps a fixed number of workers running. Use this for a persistent pool that stays available for incoming work.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: swamp-workers
spec:
  replicas: 3
  selector:
    matchLabels:
      app: swamp-worker
  template:
    metadata:
      labels:
        app: swamp-worker
    spec:
      terminationGracePeriodSeconds: 300
      containers:
        - name: worker
          image: swamp-worker
          imagePullPolicy: Never
          command: ["swamp", "worker", "connect"]
          env:
            - name: SWAMP_ORCHESTRATOR_URL
              value: wss://host.minikube.internal:9090
            - name: SWAMP_SERVER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-server-token
                  key: token
            - name: SWAMP_WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
            - name: SWAMP_WORKER_CONCURRENCY
              value: "2"
            - name: DENO_CERT
              value: /certs/ca.pem
          volumeMounts:
            - name: tls-ca
              mountPath: /certs
              readOnly: true
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
      volumes:
        - name: tls-ca
          secret:
            secretName: swamp-tls-ca

terminationGracePeriodSeconds: 300 gives in-flight steps five minutes to complete during pod termination. Kubernetes sends SIGTERM first; the worker drains its dispatches before exiting.

The memory request is 1Gi because each worker spawns a child process for dispatch execution. Under-provisioned workers are killed by the OOM killer mid-step.

If you are on Linux, replace host.minikube.internal with the gateway IP from the step above.

Apply and verify:

kubectl apply -f deployment.yaml
swamp worker list

Scale the Deployment

kubectl scale deployment swamp-workers --replicas=5

To scale down, Kubernetes terminates excess pods. Each pod drains its in-flight work during the grace period before shutting down.

kubectl scale deployment swamp-workers --replicas=1

Deploy as a Job

A Job runs a worker that processes a fixed number of dispatches and exits. Use this for batch processing where each worker handles one step and is replaced.

apiVersion: batch/v1
kind: Job
metadata:
  name: swamp-batch-worker
spec:
  completions: 5
  parallelism: 5
  template:
    spec:
      terminationGracePeriodSeconds: 300
      containers:
        - name: worker
          image: swamp-worker
          imagePullPolicy: Never
          command: ["swamp", "worker", "connect"]
          env:
            - name: SWAMP_ORCHESTRATOR_URL
              value: wss://host.minikube.internal:9090
            - name: SWAMP_SERVER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-server-token
                  key: token
            - name: SWAMP_WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: swamp-worker-token
                  key: token
            - name: SWAMP_WORKER_MAX_DISPATCHES
              value: "1"
            - name: DENO_CERT
              value: /certs/ca.pem
          volumeMounts:
            - name: tls-ca
              mountPath: /certs
              readOnly: true
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
      volumes:
        - name: tls-ca
          secret:
            secretName: swamp-tls-ca
      restartPolicy: Never

SWAMP_WORKER_MAX_DISPATCHES=1 tells the worker to drain and exit after one dispatch. Combined with restartPolicy: Never, each pod processes one step and terminates.

← PreviousRun a Fleet with Docker ComposeNext →Scale from Zero Workers