Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#969 Worker fleets phase 2 PR2: env-var config, swamp worker verify, --verify-on-enroll

Opened by stack72 · 7/4/2026· Shipped 7/5/2026

Summary

Second PR of worker fleets phase 2. Three features that make fleet provisioning smooth:

  1. Env-var configuration for `worker connect` — containers and cloud-init can't smuggle secrets through argv; every flag gains an env-var fallback
  2. `swamp worker verify` — one command proves a fleet is wired correctly end-to-end (enrollment, scheduling, capabilities, data plane)
  3. `--verify-on-enroll` — opt-in serve flag that probes each enrolling worker before it becomes schedulable, catching broken nodes at admission time

Design reference: `worker-fleets.md` at the repo root, sections "Provisioning surfaces", "Verification: `swamp worker verify`", and "Env-var configuration".

Changes

1. Env-var configuration for `worker connect`

File: `src/cli/commands/worker_connect.ts`

Add env-var fallbacks for every flag. Explicit CLI flags always win. The pattern matches the existing `SWAMP_SERVE_URL` precedent.

Variable Flag Notes
`SWAMP_ORCHESTRATOR_URL` `` positional Must start with `ws://` or `wss://`
`SWAMP_WORKER_TOKEN` `--token` Full `.` credential
`SWAMP_WORKER_LABELS` `--label` Comma-separated `k=v` pairs (e.g. `region=us-east,gpu=true`)
`SWAMP_WORKER_CACHE_DIR` `--cache-dir` Stable path for machine identity

With these set, the provisioning one-liner is just `swamp worker connect` — no arguments needed.

Implementation:

  • Read env vars in the action handler before constructing the `runWorker` options
  • Parse `SWAMP_WORKER_LABELS` as comma-separated `key=value` pairs, merged with any `--label` flags (flag wins on key collision)
  • The positional `` should become optional (not `required`) — it's required only when `SWAMP_ORCHESTRATOR_URL` is also absent
  • Update the command help text to document the env vars
  • `--token` should lose `required: true` (line 75) — required only when `SWAMP_WORKER_TOKEN` is absent

Validation: after merging flag + env var, if URL or token is still missing, throw a `UserError` naming both the flag and the env var alternative.

2. Fleet probe built-in model

New file: `src/domain/models/worker/fleet_probe_model.ts`

A built-in model whose single method exercises every seam that can break in a real deployment:

Type: `swamp/fleet-probe` Instance name: `"probe"` Method: `verify` (kind: `"action"`)

What the probe method does (inside execute):

  1. Environment check: reads `SWAMP_PROBE_MARKER` from the shipped environment snapshot — confirms env shipping works. The orchestrator sets this marker explicitly in the dispatch params.
  2. Capability round-trip: calls `context.queryData` with a trivial predicate — confirms the capability RPC channel (worker → orchestrator) works.
  3. Data-plane write + read: writes a small resource via `context.writeResource`, then reads it back via `context.readResource` — confirms the HTTP data plane (worker → orchestrator) works for both PUT and GET.
  4. Returns: `{ platform, arch, envMarker, queryOk, dataPlaneOk }` as a structured result.

Resource: `"result"` with `lifetime: "ephemeral"`, `garbageCollection: 1` (probe results are transient — only the latest matters).

Register in `src/domain/models/models.ts` barrel. Add to internal types list.

3. `swamp worker verify` CLI command

New file: `src/cli/commands/worker_verify.ts`

swamp worker verify [name] [--label k=v] [--server <url>]

Selection:

  • No args: verify all connected workers
  • Positional `name`: verify one worker by name
  • `--label k=v`: verify all workers matching the label selector

Implementation:

  1. Get the target worker list (via `workerList` from libswamp or `worker.list` protocol frame)
  2. For each selected worker, dispatch the fleet probe model to that worker via `target` placement (either via `model.method.run` over the serve protocol, or directly via `dispatchService.executeRemote` in local mode)
  3. Set `SWAMP_PROBE_MARKER=1` in the environment for each dispatch (the orchestrator's `captureEnvironment` already ships the full env — add the marker to the process env before dispatch, remove after)
  4. Collect results: pass/fail per worker with the seam that broke named in failures
  5. Render per-worker pass/fail table (log mode) or structured results (json mode)

Serve protocol frame: `worker.verify` (admin) — request carries the selection (name or labels), response carries per-worker results. Handler in `connection.ts`.

New file: `src/cli/commands/worker_verify.ts` New file: `src/libswamp/worker/verify.ts` (the libswamp operation) Register in: `src/cli/commands/worker.ts` (under the worker command group) Render in: `src/presentation/output/worker_output.ts` (`renderWorkerVerify`)

4. `--verify-on-enroll` serve flag

File: `src/cli/commands/serve.ts`

New flag:

--verify-on-enroll    Run the fleet probe on each enrolling worker before it becomes schedulable (opt-in)

File: `src/serve/worker_gateway.ts`

When `--verify-on-enroll` is active:

  1. After successful redeem + worker naming, dispatch the fleet probe to the new worker (via the dispatch service, targeting by name)
  2. If the probe passes → worker becomes schedulable normally (status: `idle`)
  3. If the probe fails → worker enters a new `unverified` status:
    • Persisted to the worker model (new `set_status` input value)
    • Visible in `swamp worker list` with the failure reason
    • Not schedulable — excluded by `eligibleWorkers`

File: `src/domain/remote/scheduler.ts`

Add `unverified` to the `SchedulableWorker.status` type and exclude it from eligibility:

status: "idle" | "busy" | "unverified";

In `eligibleWorkers`, add:

if (worker.status === "unverified") return false;

File: `src/serve/worker_gateway.ts`

Update `WorkerSnapshot` and `WorkerEntry` to support the new status. The `onWorkerEnrolled` callback should fire only AFTER the probe passes (or when `--verify-on-enroll` is off). This ensures phase 1's wake-on-enroll doesn't dispatch to an unverified worker.

File: `src/domain/models/worker/worker_model.ts`

The `set_status` method needs to accept `"unverified"` as a valid status value (and optionally a `verifyFailureReason` field).

5. Protocol version bump

Adding `unverified` to `WorkerSnapshot.status` changes the scheduling-visible type. If the `worker.list` response includes the new status value, older clients that don't expect it need to handle it gracefully. Since the status is purely orchestrator-side (the worker never sees it), no `REMOTE_PROTOCOL_VERSION` bump is needed. But the serve protocol response for `worker.list` will include the new status — clients should treat unknown statuses as "not schedulable" (graceful degradation).

Invariants to preserve

  1. Env vars are trusted values. Per CLAUDE.md security model, environment variables are not user-controlled attack vectors. No validation beyond format checking is needed.
  2. Workers stay repo-less. The probe model runs on the worker but its capabilities proxy back to the orchestrator — it doesn't need a repo, vault, or extensions on the worker.
  3. `onWorkerEnrolled` fires only for schedulable workers. With `--verify-on-enroll`, the callback must fire AFTER the probe passes, not at enrollment time. Otherwise queued steps would wake for an unverified worker and immediately re-queue.
  4. The probe is an ordinary dispatch. It goes through the normal scheduling path (target placement → dispatch service → worker → result). No special-case dispatch machinery.
  5. `--verify-on-enroll` is opt-in. Local dev doesn't want enrollment latency. The flag is deliberate v1 scope.

Tests required

Env-var config tests (`worker_connect_test.ts` or integration):

  • Env var supplies URL when positional is absent
  • Env var supplies token when `--token` is absent
  • `SWAMP_WORKER_LABELS` parsed correctly (comma-separated k=v)
  • Flag wins over env var when both set
  • Missing both flag and env var produces a clear error naming both

Probe model tests (`fleet_probe_model_test.ts`):

  • Probe reads env marker, performs query, writes/reads data plane artifact
  • Missing env marker reports the failure (env shipping broken)
  • Query failure reports the failure (capability channel broken)
  • Data-plane write failure reports the failure

Verify command tests (`worker_verify_test.ts` or integration):

  • All-worker selection dispatches to every connected worker
  • `--label` filters correctly
  • Per-worker pass/fail renders in log and json modes
  • A failed probe names the broken seam

`--verify-on-enroll` tests:

  • With flag: probe passes → worker schedulable (`idle`)
  • With flag: probe fails → worker `unverified`, excluded from scheduling, visible in `worker list`
  • Without flag: worker schedulable immediately (existing behavior)
  • `onWorkerEnrolled` fires only after probe passes

Scheduler tests:

  • `unverified` worker excluded from `eligibleWorkers`

Acceptance criteria

  1. `swamp worker connect` with only env vars set (no CLI flags) connects successfully
  2. `swamp worker verify` against a connected worker returns green (all seams pass)
  3. `swamp worker verify --label gpu=true` selects only matching workers
  4. With `--verify-on-enroll`, a worker with a broken data-plane URL lands `unverified` and never receives work
  5. With `--verify-on-enroll`, a healthy worker passes the probe and becomes schedulable
  6. `swamp worker list` shows `unverified` workers with the failure reason
  7. All existing tests pass
  8. `deno check`, `deno lint`, `deno fmt`, `deno run test`, `deno run compile` all pass

Repo conventions

  • New `.ts` files need the AGPLv3 header — run `deno run license-headers`
  • TypeScript strict, no `any`, named exports only
  • CLI commands and renderers import libswamp only via `src/libswamp/mod.ts`
  • Every new command supports both `log` and `json` output modes
  • Unit tests live next to source: `foo.ts` → `foo_test.ts`
  • Use the `ddd` skill when shaping domain changes
  • Use the `terminal-output` skill when writing render code
  • Use the `github-pr` skill for commits/PRs
  • Keep blast radius small — no adjacent refactors
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/5/2026, 12:48:41 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/4/2026, 9:34:02 PM

Sign in to post a ripple.