Skip to main content
← Back to list
01Issue
FeatureShippedExtensionsPublic
Assigneeshammz

Relationships

#2300 s3-datastore: verify conditional-write support (If-None-Match / If-Match) at setup and in doctor; a backend that ignores them makes locking silently unsafe

Opened by randybias · 9/20/2026· Shipped 9/21/2026

Problem

The S3 datastore's correctness rests on conditional writes: lock acquisition is PutObject with If-None-Match: * (s3_client.ts putObjectConditional), and the shard-first index merge uses If-Match. On an S3-compatible backend that silently ignores those headers (returns 200 instead of 412), putObjectConditional returns true for every caller: every writer "acquires" the lock, and index merges lose updates. Nothing reports this. The design doc covers a backend that answers NotImplemented (falls back and warns once) but not one that ignores the header.

Today nothing checks for it: S3DatastoreVerifier.verify() issues HeadBucket only, so swamp datastore setup, swamp datastore status and swamp doctor datastores are all green on such a backend.

Proposed solution

A conformance probe, run by datastore setup extension and doctor datastores (and cheap enough for status): under a unique probe key,

  1. PutObject with If-None-Match: * → expect success
  2. the same request again → expect PreconditionFailed (412)
  3. PutObject with a deliberately stale If-Match → expect 412
  4. delete the probe key

Fail setup (or report unhealthy) with a clear message if step 2 or 3 returns 200: "this endpoint ignores conditional writes; distributed locking would not be safe."

Alternatives considered

Documenting a manual aws s3api put-object --if-none-match '*' check in the README. Cheaper, but users of MinIO / Ceph RGW / Spaces / R2 / Hetzner are exactly the ones who will not know to run it, and the failure mode is silent corruption rather than an error.

Related: #2298 — that bug is a lock lost to a transient error; this request is about a backend on which the lock never worked at all.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.09.17.1
  • swamp: 20260916.001346.0-sha.600af419
  • OS: linux (x86_64)
  • Deno: 2.9.6
  • Shell: /bin/bash
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 8 MOREREVIEW+ 6 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/21/2026, 4:13:04 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
hammz assigned hammz9/21/2026, 2:46:18 PM
Editable. Press Enter to edit.

randybias commented 9/20/2026, 11:51:50 PM

Data point for the conformance probe, measured today against Ceph RGW 19.x (Squid):

  • PutObject + If-None-Match: * behaves correctly: first PUT 200, second PUT 412 PreconditionFailed, original body preserved (control: an unconditional PUT to the same key returns 200). Locking and crash recovery work end to end — cross-host contention serializes, and a SIGKILLed holder's lock is reclaimed after ~27 s.
  • PutObject + If-Match has a quirk: the current ETag in its quoted wire form (exactly as HeadObject returns it, and as the AWS SDK sends it) is rejected with 412, while the same ETag unquoted succeeds, as does If-Match: *. A stale ETag is correctly rejected. If-Match on GET accepts the quoted form, so it is specific to PutObject on this build.

This does not affect @swamp/s3-datastore 2026.09.17.1, whose source does not send If-Match. But design/enablers/datastores.md describes the shard index write as an If-Match compare-and-swap with a fallback only for NotImplemented. On this backend a CAS would see 412 on every attempt — indistinguishable from a lost race — so it would retry forever rather than fall back. If that CAS ships, step 3 of the probe proposed above would be worth running with a current quoted ETag as well as a stale one, and the fallback may want to treat 'my own just-read ETag was rejected' as unsupported rather than contended.

randybias commented 9/21/2026, 12:35:05 PM

Follow-up on the Ceph RGW data point above, with a correction of emphasis: treat the If-Match behaviour as likely a Ceph bug rather than something the extension must accommodate.

  • The cluster is Squid 19.2.5. Ceph tracker #73089 ("squid: rgw: conditional write doesn't work in certain scenarios") is resolved in v19.2.6 via backport PR ceph/ceph#65932 (conditional Put/Delete/MultiDelete). Neither page mentions quoted ETags specifically, so this is a lead, not a confirmed root cause — I have not yet re-tested on a fixed build.
  • Additional measurement for the lock primitive, since a sequential check does not prove mutual exclusion: 16 threads released by a barrier, each PutObject + If-None-Match: * to one fresh key through a 3-daemon HA endpoint, 12 rounds → exactly one 200 and fifteen 412s every round, stored body always the winner's (0 violations). Control: the same race without the header yields sixteen 200s. So on 19.2.5 If-None-Match: * does give mutual exclusion under contention.

If the conformance probe is built, a small concurrent variant like this is cheap and is the only form that tests the property the lock relies on.

randybias commented 9/21/2026, 12:41:14 PM

A more serious Ceph RGW result, which strengthens the case for this probe: on a versioned bucket, Squid 19.2.5 ignores If-None-Match: * on PutObject.

Measured twice independently, same endpoint, same credentials, interleaved versioned / unversioned / versioned so drift cannot explain it:

bucket versioning If-None-Match: * 1st PUT, 2nd PUT stored body
Enabled 200, 200 second writer's
never configured 200, 412 first writer's (control)

On the versioned bucket If-Match also rejects every value (current quoted, current unquoted, stale — all 412).

So on an S3-compatible backend, the same endpoint can honour conditional writes on one bucket and silently ignore them on another. @swamp/s3-datastore on a versioned bucket here would have every writer acquire the lock with no error, and datastore status stays healthy (HeadBucket only). This is exactly the case the proposed probe catches, and it argues for running it per bucket at setup, not once per endpoint. Worth a sentence in the extension README too: some older reports (e.g. #872) describe versioning as a prerequisite for conditional writes; on Ceph RGW it is the opposite.

hammz commented 9/21/2026, 4:13:16 PM

Thanks for this one, randybias — you caught a failure mode that was invisible by construction. The S3 verifier now probes both If-None-Match and If-Match under a throwaway key and reports a conditional-write verdict, so a backend that silently ignores the headers fails setup and reports unhealthy instead of passing HeadBucket while locking quietly does nothing. Your framing also surfaced a real bug underneath: putObjectConditional matched precondition failures on error name only, so an endpoint honouring If-None-Match but returning a non-XML 412 body made lock acquisition throw rather than report not-acquired. That is fixed for every caller. Shipped in swamp-extensions PR 299.

Sign in to post a ripple.