#2337 s3-datastore: since the #2300 conditional-write change, every push to Ceph RGW 19.2.5 fails (_meta.json 'kept changing'); the run still exits 0

Summary

Since the conditional-write work shipped for #2300, every push to a Ceph RGW 19.2.5 (Squid) bucket fails: the index commit gives up with [s3-sync] _meta.json kept changing under concurrent writers, although there is only one writer. The method run itself still exits 0 (see #2299), so the run's data never becomes visible to other participants and nothing says so except one [ERR] push failed line.

Measured, same bucket, same credentials, same swamp build, 4 runs each

swamp 20260922.011324.0-sha.2e949db5; unversioned bucket; single writer; fresh namespace per version.

Cause

putObjectIfMatch (datastores/_lib/s3_client.ts) sends IfMatch: etag with the ETag exactly as S3 returned it — the quoted form. As reported on #2300, RGW 19.2.5 rejects If-Match on PutObject when the current ETag is sent quoted (412), accepts the same ETag unquoted, and accepts If-Match: *. GET accepts the quoted form, so this is PutObject-specific on this Ceph build. Every compare-and-swap therefore looks like a lost race; the retry loop re-reads, gets the same ETag, fails again, and exhausts. Ceph tracker #73089 (conditional writes "in certain scenarios", fixed in 19.2.6) is a plausible root cause but not confirmed.

The conditional-write probe added in #2300 does not catch this: it sends a stale If-Match (correctly rejected) rather than the current ETag, so setup and datastore status report healthy on a backend where every commit fails.

Suggestions

1. Probe with the current quoted ETag as well as a stale one; a 412 on your own just-read ETag means If-Match is unusable here → fall back to merge-on-write (the path already used for NotImplemented) and warn, instead of retrying to exhaustion.
2. Optionally retry once with the ETag unquoted before concluding — it succeeds on this backend.
3. A push that fails its commit should not leave the command at exit 0 (#2299).

Environment: Linux x86_64, Ceph RGW 19.2.5 behind an HA endpoint, path-style addressing.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

• Extension: @swamp/s3-datastore@2026.09.22.1
• swamp: 20260922.011324.0-sha.2e949db5
• OS: linux (x86_64)
• Deno: 2.9.7
• Shell: /bin/bash