#3802 v3 bucket outage: a workflow run whose checkpoint committed is told nothing was committed, three outage errors have no stable code, and exit takes 15-60 s (found by #3756)

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished solo mode: v3 datastores on filesystem, S3 and GCS, with one writer at a time. swamp-club#3759 made a v3 command print its result only after its commit lands, and gave a commit that cannot reach the storage a readable error: datastore_v3_commit_unreachable, exit 75, "nothing this command changed was committed". After #3759 that error is often the only thing a user or a --json caller sees when the bucket is down, so it has to be true, and it has to arrive quickly.

The Phase 4 dogfood (swamp-club#3756) re-tested it on main at bc388c720, with ministack and fake-gcs behind TCP proxies that drop connections: 84 random cuts, 42 per bucket kind. The core fix holds (0 runs printed success and then lost the write, against 3 of 84 before #3759). Three gaps remain.

1. A workflow run whose checkpoint committed is told nothing was committed

When the bucket becomes unreachable during a workflow run, the command exits 75 with:

Swamp could not reach the datastore's storage (GCS getMetadata transport failure — fetch failed), so nothing this command changed was committed. Its steps had already run, so check anything they changed outside swamp before running it again once the datastore is reachable.

In 3 of 3 such runs on GCS, part of the run was committed: the run's checkpoint (swamp-club#3587) had landed before the final commit failed. The run record was left in state running with its steps pending, visible in the writer's checkout and in a fresh clone. In 1 of the 3, the first step's output was committed too.

A WRN line just before the error says "The final record of 1 workflow run(s) was not committed; run swamp workflow recover …", so the error contradicts it, and a --json caller sees only the error. Exit 75 also tells a caller a retry is safe while a running record is left behind.

workflow recover fixes it cleanly (the run goes to suspended; resume and a new run both succeed). Method runs and workflow approve don't checkpoint, and behaved as their message says.

Also seen once, right after such a run: workflow history search in the writer listed the run as succeeded with a completedAt, while workflow history get said running. A later command corrected it.

Cause. CommitUnreachableError (src/domain/datastore_v3/commit_log_unit_of_work.ts) always says nothing was committed. It doesn't know whether the root had already committed a checkpoint.

2. Three outage errors have no stable code

These printed nothing and kept nothing, and exited 1 with raw storage text:

• Lease release: S3 putObjectConditional failed key=<prefix>/_control/writer/released-000000000016 timed out after 30000ms TimeoutError — read ECONNRESET, code TimeoutError.
• Read-model catch-up on S3: Cannot apply commit 6 of partition … to the read model: S3 getObject failed key=<prefix>/_blobs/…, no code.
• Read-model catch-up on GCS: Cannot apply commit 22 of partition … to the read model: error reading a body from connection, no code.

The catch-up failures heal on their own: with the bucket back, the next command succeeds, doctor datastores passes and a fresh clone matches. No --repair is needed.

3. Exit time with the bucket down: 15–25 s, and 60 s on S3 for a saved change

Timed with a proxy that logs every refused request; the method itself ends at about 1.7 s.

datastore_v3_commit_unreachable takes about 15–25 s. On S3 (24.9 s) the phases run one after another: the commit's chunk HEAD retries (2.2–10.4 s), lease-renew retries (10.6–12.4 s), the closing Litestream sync with its 10 s deadline (12.4–22.4 s), then lease-release retries (22.8–24.8 s). On GCS it's about 15 s when the sidecar's failing sync overlaps the commit's retries, and 25 s when it runs after them.

"Saved locally, unconfirmed" (exit 75, change kept), with only replication traffic cut:

• S3: 60.8 s. The success line prints, then the closing sync waits out its full 60 s deadline while Litestream retries inside the sync.
• GCS: 0.7–1.3 s. Litestream answers POST /sync with a 500 straight away (write ltx file … connection reset).

A user on S3 waits a minute for an outcome GCS reports in a second.

Fix

1. Checkpointed runs: when a root that already committed a checkpoint then fails its final commit because the storage is unreachable, fail with its own code, for example datastore_v3_run_partly_committed, with exit 1. Its message says the run's progress up to its last checkpoint was committed, the run is left running, and to run swamp workflow recover (naming the run) once the datastore is reachable. Keep datastore_v3_commit_unreachable and exit 75 for roots where nothing landed. The --json error carries the run id. Reproduce the history search / history get mismatch and fix it here if it's in this path; if not, note it in the PR.
2. Stable codes for the other outage errors: a lease release and a read-model catch-up that fail because the storage is unreachable (the same isNetworkFailure test #3759 added) get stable codes saying the datastore was unreachable and nothing was changed, with exit 75. A catch-up failure also says the next command will finish the catch-up. Errors that aren't network failures keep their own.
3. Exit time:
   • Once a command has found the storage unreachable, don't spend time on retries that can't succeed: skip or overlap the lease renew and release retries instead of running them in series after the commit's.
   • On S3, treat Litestream's sync failing with a connection error as a definite failure, as it already is on GCS, instead of waiting out the closing sync's 60 s. The outcome is the same documented "saved locally, unconfirmed" exit 75.
   • Target: any bucket-down outcome exits within 15 s of the method ending, on both bucket kinds.

Done when

• A workflow run whose checkpoint committed, then lost the bucket, exits 1 with the new code and a message that matches what the WRN line says, in log and --json mode. Method runs and approve still get datastore_v3_commit_unreachable and exit 75.
• A lease release and a read-model catch-up cut off from the bucket each fail with a stable code and exit 75, and the next command with the bucket back succeeds without --repair.
• With the bucket down, each outcome (unreachable, unconfirmed, the new codes) exits within 15 s of the method ending on S3 and on GCS, in the dogfood's cut harness (~/dogfood-3756/cut2.sh) with the sidecar alive.
• MayLandLaterError (datastore_v3_commit_may_land, exit 1) is unchanged.
• Tests red on main first for each of these. The existing swamp-uat test "method run cut off from its bucket mid-run exits 75" passes unchanged.

Severity: medium-high. Nothing is lost, but the one message a user sees in an outage is wrong for workflow runs, three outage errors look like bugs, and an S3 user waits a minute. Blocks announcing v3 on buckets (swamp-club#3756).

Blocked by: nothing. Runs in parallel with the identity and clone-performance issues filed alongside it.