Relationships
#1760 s3-datastore: a stale machine's sync pushes its old 'latest' pointer over the newer remote one, and the rollback is permanent
Opened by david_workingmemory · 8/21/2026
Summary
Two machines sharing one @swamp/s3-datastore namespace never converge. Each data record is a directory of immutable version dirs (1/, 2/, ...) plus one mutable 2-byte latest file naming the current version, and reads resolve through latest. The version dirs are new keys so they always replicate correctly, but latest is synced by index/mtime bookkeeping with no version comparison.
Result: a machine whose local cache is behind will push its OLD latest over the newer remote one during a routine swamp datastore sync, silently rolling back what every other machine reads. The machine that had the correct value then refuses to re-push it, so the rollback is permanent.
Environment
- swamp
20260819.011806.0-sha.a9c7ee9a(both machines) @swamp/s3-datastore2026.08.12.1- Backend: Cloudflare R2 via
endpoint: https://<ACCOUNT_ID>.[HOST-1],forcePathStyle: true,region: auto, single namespace - Machine A: macOS. Machine B: Ubuntu. Identical
.swamp.yaml, samerepoId, same bucket and namespace.
Reproduction
- On machine A, run model methods that advance a record several versions. Flush with
swamp datastore sync. - Confirm the remote pointer is current by reading the object directly:
GET <bucket>/<ns>/data/<type>/<model-id>/<record>/latestreturns8. - On machine B, whose cache is behind, run a plain
swamp datastore sync. - Read the same remote object again. It now returns
6.
Machine B pushed its stale pointer over the newer remote value. No error, no warning, and the sync reports success.
Evidence
Observed on the record lease-current, where R2 held version dirs 1..8:
| Step | remote latest |
|---|---|
| after machine A flushed | 8 |
after machine B ran sync |
6 |
Machine A will not repair it. Its local latest still reads 8, but swamp datastore sync --push reports Pushed 0 files, because its index entry for that key absorbed machine B's push metadata (lastModified set to B's push time) and it therefore believes the remote is already current.
So the steady state is: remote 6, machine A local 8 and refusing to push, machine B local 6. Neither side self-corrects.
Scope at the time of diagnosis: 17 of 135 records on machine B had latest pointing below the highest version present on its own disk. The affected records were exactly those the other machine had written most recently.
Why this matters
The pointer is what every read resolves through, so a rolled-back pointer silently serves stale content for a record whose newer versions are present on disk and in the bucket. In our case one of the affected records was a mutual-exclusion lease, where a stale read means two machines can disagree about who holds it.
Additional observations, possibly the same root cause
swamp model method rundoes not flush. It leaves entries indirtyPaths, and onlyswamp datastore syncuploads them. Until the writer syncs, its most recent records are invisible to every other machine. If that is intended, it is worth documenting, because "the first swamp command that touches data will auto-hydrate" in the sharing docs led us to expect reads to be self-healing.Deleting a local
latestdoes not cause the next sync to re-fetch it, because the index still lists the key as present. It is only restored if that sync happens to have other work to do. Deleting all 138 pointers and syncing producedSync complete: 0 pulled, 0 pushed, leaving every record unreadable until we restored from a backup. A sync that verifies local presence, or a repair command, would prevent that footgun.A full re-hydration from scratch (removing the local cache entirely and letting it rebuild) does not help, since it faithfully inherits whatever stale pointer the last syncer pushed.
Suggested fix
Give the pointer write precedence rather than last-writer-wins. Either compare versions before overwriting latest and refuse to move it backwards, or derive the current version from the highest version directory present and drop the mutable pointer entirely. Conditional writes work correctly against R2 (we verified If-None-Match: * returns 200 then 412), so a compare-and-swap on the pointer is available on this backend.
Upstream repository: https://github.com/systeminit/swamp-extensions
Environment
- Extension:
@swamp/s3-datastore@2026.08.12.1 - swamp:
20260820.072114.0-sha.05d20b6a - OS:
darwin(aarch64) - Deno:
2.8.3 - Shell:
/bin/zsh
Closed
No activity in this phase yet.
stack72 commented 8/28/2026, 5:08:28 PM
Fixed in PR #239. The root cause was #1771 — pullChanged missed same-size latest pointer updates, leaving the stale value for pushChanged to overwrite the remote. With the sha256 fix in pullChanged, the local pointer is corrected before pushChanged runs, eliminating the rollback. End-to-end test confirms: pull downloads "8", push sees no diff, remote stays correct.
Sign in to post a ripple.