#3553 v3: a refused commit leaves its staged files in the checkout for good, so that checkout shows data the datastore doesn't hold (exposed by #3532; swamp-uat breakglass test failing)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). In v3 every change is a commit in commits.db, and each checkout rebuilds its files (.swamp/data/…) from the commits. A writing command stages its changes in the checkout, then commits them under the fenced writer lease (swamp-club#3359). If the commit is refused (a lost lease, version-live, version-reused) or the unit is abandoned or fails, the command reports that nothing was committed.

The bug

A v3 command whose commit is refused leaves the files it staged in its own checkout, permanently. That checkout then shows data the datastore doesn't have.

Found by swamp-uat's swamp datastore lock release --force --model mid-run loses no write that either clone reported as successful [filesystem-v3] (tests/cli/adversarial/remote_concurrency_test.ts). It has failed on every Datastore UAT run since release 20261016.021405.0-sha.b46b8895, swamp-club#3532, for example run 38934784426.

Reproduced locally (filesystem-v3, macOS) with the released binaries, 3 of 3 each way:

• 20261016.003849.0-sha.289d84ac, before #3532: passes;
• 20261016.021405.0-sha.b46b8895, #3532: fails.

What happens (instrumented run on b46b8895):

1. Clone A starts a slowRecord run, which takes the writer lease (fence 2).
2. Clone B force-releases the writer lease and the model lock, then runs and commits version 1 (fence 3) and exits 0.
3. A finishes its body, finds the lease taken over, and exits 1: "Another swamp process took over the writer lease … (fence 3) while this one held fence 2 … nothing more from this command was committed."
4. Clone A's checkout now lists versions [2, 1], latest 2, which is A's refused run. Clone B and a fresh clone list [1].
5. A later read in A doesn't clear it. A later successful write in A commits version 3, and A then lists [3, 2, 1] while B lists [3, 1]. Phantom version 2 stays in A's checkout indefinitely.

Why it showed up with #3532: #3532 is correct. Before it, A's refused write was allocated version 1, the same path as B's committed version 1, so catching up overwrote A's leftovers with B's files and hid them. #3532 allocates A above the committed high-water mark (version 2), so the leftovers sit on their own path and stay visible.

The real defects:

1. A refused, abandoned or failed v3 unit doesn't remove the files it materialised in its checkout. swamp-club#3299's interrupted-writer recovery only runs after an unclean end (its marker), not after a clean refusal.
2. The read model's catch-up never removes a checkout file that no commit names, so the leftovers persist even after later commits.
3. (Minor, follows from 1) allocation also reads the checkout's on-disk version directories, so the phantom version pushes the next write to version 3, leaving a gap in the datastore.

What to do

1. When a v3 unit's commit is refused, or the unit is abandoned or its end fails, remove the checkout files it staged that no stored commit names, and their catalog rows, before the command exits. Reuse #3299's recovery logic for exactly the unit's own staged keys rather than a whole-checkout sweep.
2. Make sure no path leaves them behind: the lease lost before commit (fenced), version-live, version-reused, a commit store error, the outbox discard paths (#3490/#3506), and serve's per-request refusal (D3).
3. Decide whether allocation should still read on-disk version directories on v3. With #3532 the commit store's high-water mark is authoritative under the lease; local directories can only add phantom numbers. If you keep them, say why.
4. Optional safety net: a cheap check at writer start that removes checkout files under data names whose versions no commit records. Only if (1) can't cover a path; keep it bounded.

Tests

• In-process integration, two checkouts on one filesystem-v3 datastore (the swamp-uat scenario):

  • A takes the lease and stages a version;
  • B breaks the lease and commits;
  • A's commit is refused;
  • assert A's checkout lists exactly what B's does, and A's next write gets B's high-water mark + 1.

  It must fail on main.

• The same for version-live and version-reused refusals, and for an abandoned unit.

• swamp-uat's test passes unchanged on filesystem-v3; it pins the user-facing behaviour.

Severity: a v3 checkout (opt-in) shows data the datastore doesn't hold, after a reported failure. It only follows a refused commit (a breakglass, a takeover, a refused version), but it's permanent once it happens. Fix before Phase 4E puts v3 on buckets, where takeovers between machines are routine.

Blocked by: nothing.