#3860 v2 S3: about 4-5% of serial workflow runs fail with datastore_unpublished_data_version ('a concurrent run or the serve poller synced') when nothing else is running (found by #3756)

What happens

On a v2 S3 datastore, about 4–5% of workflow runs fail with nothing else using the datastore:

The datastore does not hold @dogfood/blob/ version N after 3 push attempts: each time, a concurrent run or the serve poller synced the datastore over the push. The data is on this host only; run the step or method again to write and publish it.

The code is datastore_unpublished_data_version (UnpublishedDataVersionError in src/domain/datastore/datastore_sync_service.ts, the per-model lock push from swamp-club#3146).

No concurrent run and no serve existed: one script ran the workflows one after another. The message names those as the cause, so a user is sent looking for something that isn't there. The step's data stays on that host only, and the run fails.

v2 is the default format, so this hits current v2 S3 users.

Where it was seen

Found by the Phase 4 dogfood (swamp-club#3756) on ministack (an S3 emulator), not on a real S3 bucket:

• The released-era binary (ea2a36f51): 7 or more of about 150 serial workflow runs on the v2 S3 repo.
• main at bc388c720: 6 of 150 (4%). The quiet benchmark pass on main also had 2 failed samples from it, in a 5-step workflow.
• The v3 repos had no such failures.
• Not recorded: whether the v2 GCS repo showed it.

The repo was populated by the dogfood harness (~/dogfood-3756/run_populate.sh): 200 command/shell models, 40 blob models (@dogfood/blob, a local extension that writes N random bytes) and 10 @swamp/git models, with 30 workflows of 4–8 steps. Each run was a plain swamp workflow run <wf>.

Likely cause (a lead, not proven)

The dogfood's traces show a workflow's own steps running concurrently where the DAG allows it. #3146's push publishes a step's data version and then checks the remote holds it; if not, it retries, up to 3 times. Within one run, a sibling step's sync or push can write the remote index over this step's push. That's the same "another sync in the same process undid the push" case #3146 describes, but caused by the run's own steps rather than another run or serve.

Not ruled out: ministack's listing behaviour (for example, a listing that lags a recent put), which could make the "does the remote hold it?" check fail on its own.

Work

1. Reproduce in-repo, red on main: a workflow whose parallel steps each write data, on a v2 S3 datastore (in-process fake S3 or a port-0 mock), run many times. Count how often a push is undone and by which sync.
2. Rule ministack in or out: run the dogfood repro against a second S3 implementation, or with the check's listing replaced by a direct read of the key.
3. Fix the cause that 1 and 2 show. If it's sibling steps within a run, those syncs must not undo each other's pushes (for example by serialising the run's own pushes, or merging the index rather than overwriting it). If the fix belongs in the S3 extension's cache sync, it's still tracked here.
4. Fix the message: when the overwriting sync came from this process's own run, don't blame "a concurrent run or the serve poller".

Done when

• The in-repo test fails on main and passes on the fix.
• The dogfood's serial run of 150 workflow runs on a v2 S3 repo has 0 failures, against 6 on main.
• UnpublishedDataVersionError names the real cause when it's raised.
• The swamp-uat datastore suites pass unchanged.

Severity: medium-high. It's v2, the current default: about 1 workflow run in 25 fails on S3 and leaves its data on one host, with a message pointing at the wrong cause.

Blocked by: nothing. It's independent of the v3 work in flight (#3803 changes only tracing spans near this code).