Skip to main content
← Back to list
01Issue
FeatureShippedSwamp Club
Assigneesstack72

Relationships

#977 Docs: add drain, lifecycle policies, and exit-code contract to references

Opened by stack72 · 7/5/2026· Shipped 7/5/2026

Summary

Phase 3 PR1 (#976) added worker drain, --max-dispatches, --idle-timeout, signal handling, and exit-code contract. Three documentation pages need updates.

Changes needed

1. `content/manual/reference/remote-execution/worker-commands.md`

`swamp worker connect` section — add two flags to the flags table:

Flag Required Description Default
`--max-dispatches ` No Drain and exit 0 after N dispatches complete (env: `SWAMP_WORKER_MAX_DISPATCHES`) —
`--idle-timeout ` No Drain and exit 0 after being continuously idle for this duration (env: `SWAMP_WORKER_IDLE_TIMEOUT`) —

Add env vars to the env-var table (if one was added by #975):

Variable Flag
`SWAMP_WORKER_MAX_DISPATCHES` `--max-dispatches`
`SWAMP_WORKER_IDLE_TIMEOUT` `--idle-timeout`

Add examples: ```bash

Ephemeral worker: one job, then exit (k8s Job, docker run --rm)

swamp worker connect wss://orch:9090 --token --max-dispatches 1

Autoscaled: exit after 10 idle minutes so the ASG can reclaim the host

swamp worker connect wss://orch:9090 --token --idle-timeout 10m ```

`swamp worker list` section — note that the STATUS column now includes `draining` (shown in yellow) for workers that are finishing in-flight work before disconnecting.

Exit-code contract — add a new subsection (after the examples or at the end):

Exit code Meaning
0 Policy-complete: drained via signal, --max-dispatches reached, or --idle-timeout fired
Non-zero Failure: permanent enrollment rejection, unexpected error

This is what makes `restart: on-failure` in Docker Compose, `restartPolicy: Never` in k8s Jobs, and ASG health checks work correctly — platforms can distinguish "done on purpose" from "crashed."

2. `content/manual/reference/remote-execution/failure-semantics.md`

Add a new section "Graceful drain" (after "Cooperative cancellation"):

  • SIGTERM (Unix) or SIGINT triggers drain: the worker finishes in-flight work, rejects new dispatches, sends `worker.drain` to the orchestrator, disconnects, and exits 0
  • A second signal force-exits with non-zero code
  • `--max-dispatches N` and `--idle-timeout ` are programmatic drain triggers — same drain flow, same exit code
  • The orchestrator marks a draining worker as `draining` (excluded from scheduling) and treats its disconnect as deliberate — no grace window, no re-dispatch churn
  • A dispatch racing a drain is re-queued (never failed) — the `worker_draining` rejection is handled the same way as `worker_busy`

3. `content/manual/reference/remote-execution/enrollment-tokens.md`

Add a row to the Restart and reboot scenarios table:

Scenario Token state What happens
Worker drains (signal, max-dispatches, idle-timeout) enrolled Worker finishes in-flight work, disconnects cleanly, exits 0. Token remains enrolled — a new process with the same `--cache-dir` can re-enroll.

Files verified as needing NO changes

  • `reference/swamp-serve/serve-flags.md` — drain is worker-side, no new serve flags
  • `tutorials/remote-execution.md` — tutorial doesn't use lifecycle flags
  • `explanation/remote-execution.md` — conceptual, no flag-level detail
  • `reference/remote-execution/security.md` — no security implications from drain

Context

Ship alongside or shortly after #976 merges.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/5/2026, 7:33:32 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/5/2026, 7:15:05 PM

Sign in to post a ripple.