Skip to main content
← Back to list
01Issue
FeatureShippedSwamp Club
Assigneesstack72

Relationships

#1002 Worker fleets phase 7: how-to guides and explanation docs

Opened by stack72 · 7/6/2026· Shipped 7/7/2026

Summary

All fleet engine work (phases 1-6) is shipped and tested end-to-end across Docker, Docker Compose, and Kubernetes. This phase writes the user-facing documentation.

Three deliverables:

1. New explanation doc: Worker Fleets

`content/manual/explanation/worker-fleets.md`

Covers:

  • What a fleet is — a pool of workers enrolled through one fleet token, the unit of provisioning, scaling, and teardown
  • How fleet tokens differ from single-machine tokens (`--max-enrollments`)
  • Fleet member naming (`-`) and auto-injected `fleet=` labels
  • Capacity slots (`--concurrency`) and least-loaded scheduling
  • Drain and lifecycle policies (`--max-dispatches`, `--idle-timeout`, SIGTERM → drain)
  • Elastic queueing — empty pools queue instead of failing, demand is visible via `swamp worker queue`
  • How supply (workers) and demand (pending dispatches) meet in the datastore for observability

Note: vault-based token resolution for workers is a planned future enhancement (workers are currently repo-less and resolve tokens from env vars or CLI flags).

2. Update existing explanation doc

`content/manual/explanation/remote-execution.md`

Add a "Scaling with fleets" paragraph at the end of the "How the pieces connect" section, linking to the new worker-fleets explanation. The existing doc correctly describes single-worker remote execution — it doesn't need a rewrite, just an acknowledgment that fleets exist as the next level up.

3. How-to guides with inline artifacts

All tested end-to-end during development. Each guide includes the complete configuration inline (no separate `examples/` directory reference).

`content/manual/how-to/worker-fleets/docker.md` — Single Docker worker

  • Compile + build image
  • TLS cert generation (self-signed for local testing)
  • Orchestrator with `--host 0.0.0.0 --trusted-hosts host.docker.internal --auth-mode token`
  • Worker via env vars: `SWAMP_ORCHESTRATOR_URL`, `SWAMP_SERVER_TOKEN`, `SWAMP_WORKER_TOKEN`, `SWAMP_WORKER_LABELS`, `DENO_CERT`
  • Verify + run placed workflow

`content/manual/how-to/worker-fleets/docker-compose.md` — Docker Compose fleet

  • Fleet token with `--max-enrollments`
  • `docker-compose.yaml` with env var config, `SWAMP_WORKER_CONCURRENCY`, `stop_grace_period: 5m`, `restart: on-failure`
  • Scaling with `docker compose --scale worker=N`
  • Fan-out workflow demonstrating parallel execution across workers
  • Note: steps targeting the same model definition serialize on the per-model lock; use distinct model names (e.g., `modelName: worker-${{ self.item }}`) for true parallelism

`content/manual/how-to/worker-fleets/kubernetes.md` — Kubernetes fleet (minikube)

  • Loading image into minikube
  • Secrets for worker token, server token, and TLS CA cert
  • Deployment manifest: `terminationGracePeriodSeconds: 300`, `SWAMP_WORKER_CONCURRENCY`, resource requests (note: 1Gi memory minimum for worker + dispatch runner child process)
  • Job manifest: `--max-dispatches 1`, `restartPolicy: Never`
  • Scaling the Deployment up/down, observing drain behavior
  • Finding the orchestrator URL from inside minikube (`host.minikube.internal` on macOS/Windows, gateway IP on Linux)

`content/manual/how-to/worker-fleets/scale-from-zero.md` — Scale-from-zero walkthrough

  • Empty pool → submit workflow → step queues (`step_queued` event visible) → connect a worker → step dispatches immediately (wake-on-enroll) → run completes
  • Demonstrates elastic queueing, `swamp worker queue`, and the queue timeout

Key learnings from testing to document

  • `--trusted-hosts` is required for Docker/k8s deployments (off-loopback Host header validation)
  • `SWAMP_SERVER_TOKEN` is required for `--auth-mode token` deployments (workers need server-level auth in addition to enrollment tokens)
  • `DENO_CERT` env var trusts self-signed certs (not a swamp flag — a Deno runtime setting)
  • Worker + dispatch runner child process needs ~1Gi memory in container environments
  • Per-model datastore lock serializes steps writing to the same model definition — use distinct model names in forEach for parallelism
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/7/2026, 12:23:06 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/6/2026, 11:52:06 PM

Sign in to post a ripple.