#3359 Datastore rework Phase 4: a fenced writer lease for v3 datastores, replacing Phase 3's writer lock

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits.

Phases 1–3 are done (main at fc312a85). v3 works today for new, empty filesystem datastores with a single writer (swamp-club#3279, and the fixes #3299, #3300, #3301, #3337). It's tested in CI by swamp-uat's filesystem-v3 backend.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream continuously replicates it to the S3 or GCS bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease in the bucket's control plane makes sure only one machine writes.

The decisions so far are in design/enablers/datastore-litestream.md (swamp-club#3234) and in design/enablers/datastores.md, under the "v3 …" sections.

Phase 4 split (all on #2865's plan):

• A: real-bucket latency spike.
• B: Litestream binary management.
• C: fenced writer lease.
• D: Litestream sidecar and replication (needs A, B and C).
• E: v3 on S3 and GCS (needs D, plus the extensions' storage/1 and swamp-club#3204).
• F: durability leftovers.
• G: extension work.
• H: swamp-uat s3-v3 and gcs-v3.
• I: dogfood and benchmark.

A, B and C run in parallel.

A decision for Phase 4: a v3 write command waits for Litestream to confirm replication (POST /sync {"wait": true}) before it reports success. One sync covers every commit the command made. The design promises that no acknowledged write is lost, and a dead laptop must not take acknowledged writes with it. Read-only commands never wait.

Goal

Replace Phase 3's v3 writer lock with a fenced writer lease that works on a filesystem datastore now and on S3 and GCS buckets in Phase 4. Every commit names the fence its writer holds, so a writer that lost its lease (a zombie) can never land a commit.

Today (Phase 3)

src/infrastructure/datastore_v3/v3_writer_lock.ts is a FileLock at <datastore>/_control/writer.lock, held by writing contexts and by serve for its lifetime:

• across hosts (#3279's sameHostPidCheck), another host's holder is judged by TTL alone;
• no fence: a holder paused past its TTL keeps committing after another takes over (documented as Phase 3's limit);
• commits carry no fence.

The domain already enforces fences. Partition.append (swamp-club#3233):

• refuses a commit whose fence is below the partition's highest (stale-fence);
• refuses an unfenced commit once the partition has seen a fence (unfenced).

The Litestream record also decides that the replica path is per fence (_litestream/<fence>/), so a fence must exist before D can replicate.

Design

1. A lease port (domain), with acquire (taking over an expired holder), renew (the heartbeat), release and inspect. A held lease has a fence: an integer that only ever increases, one per acquisition, never reused.

2. No compare-and-swap is needed. ControlPlaneStore offers put, putIfAbsent, get, delete and list; it has no conditional update. Build the fence on create-only keys:

   • acquiring fence n+1 is putIfAbsent(_control/writer/fence-<n+1>), holding the holder identity and expiry;
   • exactly one of two racing acquirers wins;
   • a holder renews by writing its own heartbeat record;
   • on every renewal, and before every commit's sync confirmation, it checks that no higher fence key exists. If one does, it has been fenced: it stops writing and fails the command clearly.

   Record the scheme, the clock-skew bound and the expiry rule in datastores.md. If you find a better primitive in the store, say why.

3. Two adapters:

   • Filesystem datastores: files under <datastore>/_control/writer/, created exclusively (createNew, or link with a fallback, as #3279's marker writer does). This replaces the FileLock.
   • Control plane (S3, GCS, extensions): over ControlPlaneStore.putIfAbsent and get. Test it on the in-memory remote. The extensions already offer putIfAbsent; anything missing is filed for G, not built here.

4. Commits carry the fence.

   • The commit-log unit (commit_log_unit_of_work.ts, runInCommitLogRoot's fence option) stamps every commit with the holder's fence.
   • The commit store's Partition.append already refuses stale or unfenced commits once a fence is seen.
   • Existing v3 filesystem datastores have unfenced history. The first fenced commit simply starts fencing, which the domain already allows. Test that an existing Phase 3 datastore upgrades cleanly.

5. Behaviour users see:

   • one writer at a time, as today: the second is refused with datastore_v3_writer_busy, with the same message and code;
   • a crashed holder's lease expires, and the next writer takes over with fence n+1;
   • a zombie (a writer paused past expiry that wakes up after a takeover) gets a clear error on its next commit, and nothing it does lands;
   • swamp datastore lock status|release report and release the lease (with the same refusal rules as today);
   • serve holds and renews the lease for its lifetime.

6. Litestream. Expose the current fence so D can replicate to _litestream/<fence>/. D does the replication.

7. v2 doesn't change. FileLock keeps its v2 behaviour, and v2 never uses the lease. Apply the v2 gate wherever shared code changes (#3240, #3250, #3279): suites unedited, and the route recorder identical.

Tests

• Property or model-based tests for the lease (fast-check), over random interleavings of acquire, renew, expire, take over and commit by two or three writers:
  • no two holders ever have the same fence;
  • a commit by a non-current holder never lands.
• Both adapters: filesystem on a real temp directory, and the in-memory remote's control plane. Cover a racing acquire, expiry and takeover, a zombie's commit refused, release, and inspect.
• Upgrade: a Phase 3 datastore with unfenced commits works, and its first fenced commit starts fencing.
• CLI and serve on filesystem-v3, in-process: busy refusal, lock status and release, serve holding the lease, and a crashed holder taken over.
• swamp-uat#623's lock and takeover journeys keep passing against the release. If a message or code has to change, say so in the PR so swamp-uat can follow.
• Injected clocks only. No sleeps.

Out of scope

• Litestream (B and D), the bucket opt-in (E) and team mode's leader lease (Phase 5).
• Phase 5 builds on this lease for the serve leader. Keep the port general enough, but don't build the leader.

Blocked by: nothing. It runs in parallel with A and B.