Relationships
#4431 Team mode: a member's command ends honestly when serve refuses, loses its lease or is unreachable (exit contract, F18, F7, F2, F26, F15 from #4374)
Opened by stack72 · 11/5/2026· Shipped 11/5/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), run 3, release 20261105.030003.0 (contains #4387 and #4401). Code references are at 363a1dad. This is one of the two pieces of work that gate dogfood run 4 (target: shipped by Mon 9 Nov).
A team member's command talks to serve through the serve datastore client. When that conversation goes wrong (a lease lapses, serve refuses a commit, serve is unreachable), the command must end with an honest, coded result. Today it can report success for a failure, hang for 12 minutes, or print a raw stack trace.
Ship in two PRs
PR 1 (unblocked, ship first): the exit contract
- A cancelled run exits 0. The workflow run renderer marks the run failed on a
cancelledevent and on acompletedevent with statusfailed, but not on acompletedevent with statuscancelled(src/presentation/renderers/workflow_run.ts~730 and ~1305). Scripts and CI read a dropped output as success. Every run that endscancelledorfailedmust exit non-zero, on every datastore, not only v3. - F18: a refused commit reaches the member as a raw error. Serve answers 409
commit_lease-lost(src/serve/datastore_api.ts~1010), the client drops the code, and the member sees exit 1 with a stack trace fromrefusedError(src/infrastructure/datastore_v3/commit_log_unit_of_work.ts~645). A fenced-out writer ("Another swamp process took over the writer lease … fence 35 while this one held fence 34") also exits 1. Both must be a coded, retryable error: exit 75, a stable code, no stack trace. The code name mixes_and-; keep the old spelling accepted if anything matches on it. - F7: a refused edit prints
"status":"updated". Aconfig_edit_conflictrefusal exits 1 but prints"status":"updated"on stdout (intermittent). A refusal must never print a success status. - F2: a stack trace in
--jsonfor an unknown subcommand. JSON mode prints a structured error only.
PR 2: the member's run under a lost lease or a lost serve
- F26: a long step loses its model lease. A step producing a 64 MiB output (~50 s) has its writes refused at commit: "Serve refused this step's writes while it held the behavior's lock: the lease may have lapsed" (
src/cli/repo_context.ts~2623). The output is never stored and the run endscancelled(and exits 0, item 1). Seen in 2 of 6 upload runs in run 3, 3 of 7 across runs 2–3, and it once left a 35-byte stored version.- The client already renews the lease: TTL 30 s, a renewal every 10 s (
src/infrastructure/datastore_v3/serve_lock.ts~44 and ~478). So three renewals in a row failed, and only during large uploads. Do not add another timer: find why the renewals failed. - Candidates, unconfirmed: renewals queued behind the upload on the same connection through the TLS load balancer, or the client's or serve's event loop busy encoding the 64 MiB output.
- Start from the dogfood's evidence for run
acfe2f15(set 2, 09:16 UTC): the "Could not renew the model lease" warnings and the renewal spans. If the cause turns out to be on serve's side, coordinate with the serve start/stop/replication issue.
- The client already renews the lease: TTL 30 s, a renewal every 10 s (
- F15: a member's run does not stop when serve is unreachable. With a verified network cut, a running workflow exits only after 711–782 s, with exit 1 ("Swamp lost its connection to serve … could not record that this change must not be committed"), holding the checkout's writer lock throughout (
recovergetsdatastore_v3_writer_busy). 9 of 9 rounds across runs 2–3, unchanged by #4401. The behaviour was decided in the architecture review:- calls to serve get deadlines (
AbortSignaltimeouts); - a run with no lease covering its writes stops at its next commit with exit 75, releases the writer, and leaves its steps recoverable;
- only a run holding an explicit lease may continue offline and queue its writes (today's queue-and-flush behaviour, which works).
- calls to serve get deadlines (
Must not regress
- No acknowledged write lost (0 across 18 failover rounds, concurrent writers and offline flush).
- Failover write pause 4–7 s.
- Offline: a definition edit refused with exit 75
datastore_v3_offline_config_edit; writes under an explicit lease queue and flush. - Leases, admin force-release, cancel, kill-and-recover (
unknownplus--acknowledge-unknown).
Done when (dogfood run 4 checks)
m_big(17 MiB and 64 MiB outputs in one workflow, at the start and end of each set) passes 6 of 6, with both outputs read back.- A forced lease expiry mid-step gives exit 75 with a code and no stack trace.
- A run that ends
cancelledorfailedexits non-zero. c09(cut serve once the step's command is running, hold up to 30 min) exits 75 within 15 s, andswamp workflow recover --assess-onlyworks straight after.- No
"status":"updated"on any refusal inr02.
In-repo tests for each item: the exit contract with the renderer and CLI tests; F26 with a red-checked test that starves renewals during a large output; F15 with a stalled-serve test that asserts the deadline, exit 75 and a released writer.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.