Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4431 Team mode: a member's command ends honestly when serve refuses, loses its lease or is unreachable (exit contract, F18, F7, F2, F26, F15 from #4374)

Opened by stack72 · 11/5/2026· Shipped 11/5/2026

Found by the team-mode dogfood on real GCS (swamp-club#4374), run 3, release 20261105.030003.0 (contains #4387 and #4401). Code references are at 363a1dad. This is one of the two pieces of work that gate dogfood run 4 (target: shipped by Mon 9 Nov).

A team member's command talks to serve through the serve datastore client. When that conversation goes wrong (a lease lapses, serve refuses a commit, serve is unreachable), the command must end with an honest, coded result. Today it can report success for a failure, hang for 12 minutes, or print a raw stack trace.

Ship in two PRs

PR 1 (unblocked, ship first): the exit contract

  1. A cancelled run exits 0. The workflow run renderer marks the run failed on a cancelled event and on a completed event with status failed, but not on a completed event with status cancelled (src/presentation/renderers/workflow_run.ts ~730 and ~1305). Scripts and CI read a dropped output as success. Every run that ends cancelled or failed must exit non-zero, on every datastore, not only v3.
  2. F18: a refused commit reaches the member as a raw error. Serve answers 409 commit_lease-lost (src/serve/datastore_api.ts ~1010), the client drops the code, and the member sees exit 1 with a stack trace from refusedError (src/infrastructure/datastore_v3/commit_log_unit_of_work.ts ~645). A fenced-out writer ("Another swamp process took over the writer lease … fence 35 while this one held fence 34") also exits 1. Both must be a coded, retryable error: exit 75, a stable code, no stack trace. The code name mixes _ and -; keep the old spelling accepted if anything matches on it.
  3. F7: a refused edit prints "status":"updated". A config_edit_conflict refusal exits 1 but prints "status":"updated" on stdout (intermittent). A refusal must never print a success status.
  4. F2: a stack trace in --json for an unknown subcommand. JSON mode prints a structured error only.

PR 2: the member's run under a lost lease or a lost serve

  1. F26: a long step loses its model lease. A step producing a 64 MiB output (~50 s) has its writes refused at commit: "Serve refused this step's writes while it held the behavior's lock: the lease may have lapsed" (src/cli/repo_context.ts ~2623). The output is never stored and the run ends cancelled (and exits 0, item 1). Seen in 2 of 6 upload runs in run 3, 3 of 7 across runs 2–3, and it once left a 35-byte stored version.
    • The client already renews the lease: TTL 30 s, a renewal every 10 s (src/infrastructure/datastore_v3/serve_lock.ts ~44 and ~478). So three renewals in a row failed, and only during large uploads. Do not add another timer: find why the renewals failed.
    • Candidates, unconfirmed: renewals queued behind the upload on the same connection through the TLS load balancer, or the client's or serve's event loop busy encoding the 64 MiB output.
    • Start from the dogfood's evidence for run acfe2f15 (set 2, 09:16 UTC): the "Could not renew the model lease" warnings and the renewal spans. If the cause turns out to be on serve's side, coordinate with the serve start/stop/replication issue.
  2. F15: a member's run does not stop when serve is unreachable. With a verified network cut, a running workflow exits only after 711–782 s, with exit 1 ("Swamp lost its connection to serve … could not record that this change must not be committed"), holding the checkout's writer lock throughout (recover gets datastore_v3_writer_busy). 9 of 9 rounds across runs 2–3, unchanged by #4401. The behaviour was decided in the architecture review:
    • calls to serve get deadlines (AbortSignal timeouts);
    • a run with no lease covering its writes stops at its next commit with exit 75, releases the writer, and leaves its steps recoverable;
    • only a run holding an explicit lease may continue offline and queue its writes (today's queue-and-flush behaviour, which works).

Must not regress

  • No acknowledged write lost (0 across 18 failover rounds, concurrent writers and offline flush).
  • Failover write pause 4–7 s.
  • Offline: a definition edit refused with exit 75 datastore_v3_offline_config_edit; writes under an explicit lease queue and flush.
  • Leases, admin force-release, cancel, kill-and-recover (unknown plus --acknowledge-unknown).

Done when (dogfood run 4 checks)

  • m_big (17 MiB and 64 MiB outputs in one workflow, at the start and end of each set) passes 6 of 6, with both outputs read back.
  • A forced lease expiry mid-step gives exit 75 with a code and no stack trace.
  • A run that ends cancelled or failed exits non-zero.
  • c09 (cut serve once the step's command is running, hold up to 30 min) exits 75 within 15 s, and swamp workflow recover --assess-only works straight after.
  • No "status":"updated" on any refusal in r02.

In-repo tests for each item: the exit contract with the renderer and CLI tests; F26 with a red-checked test that starves renewals during a large output; F15 with a stalled-serve test that asserts the deadline, exit 75 and a released writer.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 11 MOREREVIEW+ 16 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/5/2026, 11:05:09 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/5/2026, 4:32:22 PM

Sign in to post a ripple.