#3759 v3 on S3/GCS: a write whose commit cannot reach the bucket prints "succeeded" on --json, exits 1, and is lost (confirm-before-output, found by #3756)

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished solo mode: v3 datastores on filesystem, S3 and GCS, with one writer at a time. A v3 command runs inside a root unit of work, which commits its changes to the commit log when the command ends. The command currently renders its output inside that root, before the commit. The design doc already lists one consequence as a known limit, "confirm-before-output (9a)": --json can print success and then exit 75 when replication fails at close, with the change kept locally and replicated by the next write.

Dogfooding solo v3 (swamp-club#3756) found a worse consequence of the same cause.

What happened

On a v3 bucket datastore (S3 or GCS), a write command whose commit cannot reach the bucket prints a success document on --json stdout, then exits 1. The write is not kept anywhere: it is absent from the writer's own checkout, absent from a fresh clone after the bucket is back, and the commit outbox is empty.

This goes past the known limit in design/enablers/datastores.md (a success document followed by exit 75 when replication fails at close, with the change saved locally and replicated later). Here the success document is followed by exit 1 and the change is lost, so a --json consumer that trusts stdout records a write that does not exist.

Found while dogfooding solo v3 (swamp-club#3756), binary built from main at ea2a36f51, on ministack (S3) and fake-gcs (GCS), with a TCP proxy in front of each that can drop every connection.

Steps to reproduce

1. A v3 datastore on a bucket: swamp datastore setup extension @swamp/s3-datastore --config <bucket, new prefix, endpoint through the proxy> --format 3.
2. A behavior whose method writes a file output of a few MB, then waits about 1.5 s before returning.
3. Run it with --json; as soon as the method has written its content, make the bucket unreachable, and keep it unreachable until the command exits.

Observed (2 of 2 on S3, 2 of 2 on GCS):

• stdout: "status": "succeeded" with an outputId.
• stderr: S3 headObject failed key=<prefix>/_blobs/_default/sha256/... timed out after 30000ms TimeoutError; GCS getMetadata transport failure — fetch failed.
• exit code 1.
• swamp data get in the same checkout, and in a fresh clone after the bucket is back: not found.

With a random cut point (20 method runs and 20 workflow runs per bucket kind), this happened 2 times on S3 (method runs) and 3 times on GCS (1 method run, 2 workflow runs), and the documented exit-75 case happened 0 times.

For comparison:

• If the bucket comes back before the commit starts, the write is kept and the command exits 0 (2 of 2).
• If the bucket drops just after the success line, the documented case occurs: exit 75, "saved locally, but swamp could not confirm…", and the next command replicates it (3 of 3).

Expected

Either the command does not print success for a write it then drops (it fails with no success document), or the commit is kept locally like the exit-75 case and replicated once the bucket is back.

Notes

This shares the cause with the confirm-before-output limit (the command renders inside its root unit of work, before the unit commits). It bears on the swamp-club#3756 decision about announcing v3 on buckets.

What to do

Recommended: fix 9a itself. A v3 command renders its result only after its root has committed. Then a command never prints success for a write that didn't land:

• a commit that fails prints only the error (exit 1);
• a commit that lands but can't be confirmed replicated prints its success and then the exit-75 warning, as today.

Options, in order of preference; say which you chose:

1. Render after the root ends: the use case returns its result, the root commits, and only then does the presenter write stdout. Check the --json path and log mode both.
2. A buffered presenter for v3: hold stdout until the root has committed; discard it and print the error if the commit fails.

Not recommended: keeping the failed commit locally, like the exit-75 case, and replicating it later. The failure here happens before the commit lands (a chunk check that can't reach the bucket). Landing it later would change what "the command failed" means, and would need its own recovery semantics.

Scope: every v3 write command and serve's request handlers. Workflow runs too: 2 of the GCS cases were workflow runs. Find where commands render inside runInCommitLogRoot / withUnitOfWork, and cover them all, with a fitness rule if there's a pattern to pin.

Docs: update the 9a known limit in design/enablers/datastores.md. Once this lands, the remaining limit is only "success, then exit 75, change kept and replicated by the next write".

Tests

Each must fail on main first.

• A cut before the commit: a write whose commit can't reach the bucket prints no success document on --json stdout, exits non-zero, and keeps nothing. Use an in-process storage that fails the pre-append chunk check.
• A cut after the commit, when replication can't be confirmed: success, then exit 75, and the change is kept (the existing behaviour, unchanged).
• The same for a workflow run.
• Log mode: no success line before a failed commit.
• swamp-uat passes unchanged. If a swamp-uat test pins output order, list it in the PR.

Severity: high. A --json consumer (an agent, CI) records a write that doesn't exist. It blocks announcing v3 on buckets (swamp-club#3756).