Relationships
#3902 v3 crash recovery's progress line breaks --json stderr: a failing command that recovered prints text before its JSON error (regression from #3363 / #3893)
Opened by stack72 · 10/25/2026
Description
swamp#3363 (fixing #3893) has a v3 checkout's crash recovery print system │ Recovering work an interrupted command left in this checkout… on stderr in both output modes, through createLockProgressWriter("system"), which checks only -q. That choice is recorded in design/enablers/datastores.md.
It conflicts with an existing rule in design/surfaces/auth-gate.md (~line 106): "JSON mode has no console log output, and a failing command's stderr must stay one JSON document, so the warning is held until the run ends." #3363 does not mention that rule.
So a --json command that recovers an interrupted writer and then fails writes a text line followed by its JSON error document to stderr, and the stderr no longer parses as JSON:
system │ Recovering work an interrupted command left in this checkout…
{"error": "Another swamp process took over the writer lease ...", "code": "datastore_v3_writer_fenced", ...}Found by swamp-uat
Run: https://github.com/swamp-club/swamp-uat/actions/runs/40149814398, swamp 20261025.010734.0-sha.02a2b20b (#3363's merge commit). The failure is the same on s3-v3 and gcs-v3:
tests/cli/datastore/v3_datastore_test.ts: "swamp takes over the v3 writer lease of a writer on another machine only after its TTL, and a writer stopped there is fenced when it resumes", step "a writer stopped on another machine"
error: SyntaxError: Unexpected token 's', " system │ R"... is not valid JSON
const refusal = RefusalSchema.parse(JSON.parse(result.stderr));Steps to reproduce
- Use a v3 datastore (S3 or GCS) with two checkouts, A and B.
- In B, start
swamp model method run <slow-model> execute --jsonand SIGKILL it while it holds the writer lease. This leaveswriter.jsonin B's checkout. - In B, start
swamp model method run <slow-model> execute --jsonagain. It recovers first, which prints the line to stderr. Stop it (SIGSTOP) mid-body. - In A, run a write and let it wait out the lease TTL and take over.
- Resume B. It is fenced and exits 1.
- B's stderr is the recovery line followed by the
datastore_v3_writer_fencedJSON document, andJSON.parse(stderr)fails.
More generally: any --json command that runs an interrupted-writer recovery and then fails.
Expected
In --json mode, a failing command's stderr is one JSON document. Either the recovery progress line does not go to stderr in JSON mode, or it is held the way the auth-gate warning is. Log mode keeps the line, which is what #3893 wanted; -q still silences it; --json stdout is unchanged. Update datastores.md to say "in log mode".
The lock and sync progress lines use the same writer and also ignore the output mode, so they can break the same rule when a --json command waits on a lock and then fails. A fix at the writer level would cover both.
Impact
Low today: v3 datastores are not GA, so no user can reach this. It should be fixed before v3 GA, because agents and scripts driving swamp with --json against long-lived checkouts parse the error code from stderr. Exit codes and data are unaffected. Until then, the s3-v3 and gcs-v3 UAT shards fail on every release run.
Environment
- swamp
20261025.010734.0-sha.02a2b20b(Latest release) - swamp-uat datastore matrix, s3-v3 and gcs-v3 backends, Linux CI runner
Open
No activity in this phase yet.
Sign in to post a ripple.