#3882 v3 bucket outage: a command cut off while becoming the writer says its changes are saved locally when nothing was saved (workflow approve), and the suggested recover fails for a run cut mid-step (found by #3756)

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished solo mode. swamp-club#3802 gave every v3 bucket outage a stable code that says what happened. The Phase 4 dogfood (swamp-club#3756) re-ran its cut tests on release 20261024.011325.0 (which includes #3801, #3802 and #3803) with ministack and fake-gcs behind proxies that drop connections. Most of the bar was met: no write was lost after a printed success, no failure was raw or uncoded (17 were before), and every failure exited within 8.3 s. Two outage messages still don't match what happened.

1. "Saved locally" when nothing was saved

What happens. When the bucket drops while a command is becoming the writer, before it has written anything, the command exits 75 with datastore_v3_replication_unconfirmed:

This command's changes are saved locally, but swamp could not confirm that they reached the datastore's replica: the replication lineage of this writer could not be recorded: … They are kept in commits.db and replicate when the next write command runs in this checkout …

Nothing was saved. In the re-run, 5 of 6 such GCS outcomes and 1 on S3 (cuts 0.54–0.83 s into the command) kept nothing:

• 3 workflow runs: no run record exists; the runs never started.
• 2 workflow approves: commits.db has no approval commit, and the gate is still waiting_approval. An operator told their approval was saved won't retry it, and the workflow stays blocked.

On S3, the six outcomes whose cut landed later (0.68–1.64 s) did keep their change, so the message is right there.

Cause. In src/infrastructure/datastore_v3/litestream/v3_copy.ts (around line 851), any failure while the writer records its replication lineage is wrapped in V3ReplicationError. That happens as the command becomes the writer, before its first commit. V3ReplicationError always says the command's changes are saved locally.

Fix. datastore_v3_replication_unconfirmed is only for a command that committed something this run, i.e. its commit log moved during the command. When nothing was committed:

• a network failure (isNetworkFailure) gives datastore_v3_storage_unreachable (exit 75, nothing changed), as #3802 does for the rest of the writer's start;
• any other failure keeps its own error, worded as nothing changed.

Apply the rule wherever V3ReplicationError is thrown, not only in v3_copy.ts: there are 13 throw sites across v3_copy.ts, litestream_replicator.ts, v3_repo_context.ts and replication_group_commit.ts. Prefer one decision at the boundary, from whether this command committed, over 13 local ones.

2. The recover command it suggests fails when the cut lands mid-step

What happens. datastore_v3_run_partly_committed names swamp workflow recover <workflowId> --run <runId>. When the run was cut while a step was running, that command exits 1 in 7 of 7 runs:

Cannot auto-recover: 1 unknown step(s) lack guard expressions … Use --acknowledge-unknown

With --acknowledge-unknown added, the run recovers to suspended, resume succeeds, and history get and history search agree. When the cut lands between steps, the command works as given.

Fix. When a run the error names has a step whose outcome is unknown (it was running when the run was cut), the message says so: check what that step did outside swamp, then run swamp workflow recover <workflowId> --run <runId> --acknowledge-unknown. Runs with no such step keep today's command. --json adds, per run, whether it needs the flag.

Done when

• A workflow approve and a workflow run cut while becoming the writer fail with datastore_v3_storage_unreachable (exit 75), and say nothing was changed, on S3 and GCS. A command cut after its commit still gets replication_unconfirmed with its change kept.
• A run cut mid-step gets a message whose recover command works as printed. A run cut between steps is unchanged.
• Tests fail on main first for each case, through the fault seams #3759 and #3802 added.
• The dogfood's cut harness (~/dogfood-3756/cut2.sh, release vs branch) shows no "saved locally" outcome that kept nothing, and every printed recover command works.
• MayLandLaterError and the other #3802 codes are unchanged. The swamp-uat datastore suites pass unchanged.

Severity: medium-high. Nothing is lost, but the error tells an operator an approval was saved when it wasn't, so the workflow stays blocked. Blocks announcing v3 on buckets (swamp-club#3756).

Blocked by: nothing.