Relationships
#2192 Ephemeral worker fleets accumulate worker records and token bindings that are never reaped
Opened by keeb · 9/16/2026· Shipped 9/17/2026
Summary
Worker fleets built from ephemeral pods (Kubernetes Deployments/Jobs without a persistent --cache-dir) accumulate state that is never cleaned up:
- Every new pod presents a new machine-id, so each one appends a binding to the fleet token's
bindingsarray and creates a newworker-<token>-<suffix>worker record. - Both the
swamp/enrollment-tokenandswamp/workerresources are declaredlifetime: "infinite", soswamp data gcnever expires them;data gc --dry-runreports 0 entries. - There is no verb to prune disconnected workers or stale bindings.
The Kubernetes how-to recommends exactly this deployment shape (Deployment + fleet token, no persistent cache dir), so the growth is the default outcome. In a short test (a handful of pod replacements and one rotation) the repo ended with 9 worker records, 7 of them disconnected; a production fleet with autoscaling churns far faster. The bindings array also grows without bound on an --max-enrollments unlimited token, and each enrollment rewrites that record.
Observed on 20260911.215321 (d18f2d86); lifetimes are unchanged at 34d98319.
Where
src/domain/models/worker/worker_model.ts— resourcelifetime: "infinite".src/domain/models/worker/enrollment_token_model.ts— resourcelifetime: "infinite",bindingsappend on each new machine.src/serve/worker_gateway.ts— enrollment path creating per-machine worker names for fleet tokens.
Fix approach
Give disconnected worker records a reaping policy (e.g. remove after a configurable time disconnected, or when their enrollment token is expired/revoked), and drop fleet bindings that belong to reaped workers or expired tokens. Expose it through swamp data gc / swamp run gc or a swamp worker prune verb, and document the recommended settings for ephemeral fleets in the worker-fleets Kubernetes guide.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.