Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#2192 Ephemeral worker fleets accumulate worker records and token bindings that are never reaped

Opened by keeb · 9/16/2026· Shipped 9/17/2026

Summary

Worker fleets built from ephemeral pods (Kubernetes Deployments/Jobs without a persistent --cache-dir) accumulate state that is never cleaned up:

  • Every new pod presents a new machine-id, so each one appends a binding to the fleet token's bindings array and creates a new worker-<token>-<suffix> worker record.
  • Both the swamp/enrollment-token and swamp/worker resources are declared lifetime: "infinite", so swamp data gc never expires them; data gc --dry-run reports 0 entries.
  • There is no verb to prune disconnected workers or stale bindings.

The Kubernetes how-to recommends exactly this deployment shape (Deployment + fleet token, no persistent cache dir), so the growth is the default outcome. In a short test (a handful of pod replacements and one rotation) the repo ended with 9 worker records, 7 of them disconnected; a production fleet with autoscaling churns far faster. The bindings array also grows without bound on an --max-enrollments unlimited token, and each enrollment rewrites that record.

Observed on 20260911.215321 (d18f2d86); lifetimes are unchanged at 34d98319.

Where

  • src/domain/models/worker/worker_model.ts — resource lifetime: "infinite".
  • src/domain/models/worker/enrollment_token_model.ts — resource lifetime: "infinite", bindings append on each new machine.
  • src/serve/worker_gateway.ts — enrollment path creating per-machine worker names for fleet tokens.

Fix approach

Give disconnected worker records a reaping policy (e.g. remove after a configurable time disconnected, or when their enrollment token is expired/revoked), and drop fleet bindings that belong to reaped workers or expired tokens. Expose it through swamp data gc / swamp run gc or a swamp worker prune verb, and document the recommended settings for ephemeral fleets in the worker-fleets Kubernetes guide.

02Bog Flow
OPENTRIAGEDIN PROGRESSSHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 15 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/17/2026, 11:41:05 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/17/2026, 8:57:57 PM

Sign in to post a ripple.