#3565 v3 replication: check a possible S1 in serve's retake, fix serve wedging on a retake mid-replacement, and stop 'the next write replicates them' being false (bug hunt group 2)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). While a v3 datastore replicates, each checkout keeps its own copy of commits.db and a Litestream sidecar replicates it (swamp-club#3416, #3482). A writer makes its copy current under the writer lease (restoring from the published lineage when another checkout wrote), and serve replaces its sidecar and retakes the lease in the background (D2/D3, #3482, #3489).

On main replication is reachable only through a test-only setting (SWAMP_V3_REPLICA). Phase 4E (swamp-club#3534, PR #3180) switches it on for every v3 datastore on S3 and GCS, so the findings below go live when it merges. A v3 bug hunt (2026-10-16, verified on main d3b76490c) found four. Line numbers below are from PR #3180's branch (5f4441c03) where E moved the code; build on whichever is on main when you start (#3180 is expected to merge first).

First: check a possible S1 (V3-54's neighbour)

Check this before anything else, and report the result on this issue. Serve's sidecar replacement (replace() in v3_repo_context.ts, :1195-1243 on the branch) calls ensureV3CopyCurrent(repoDir, config, "write"), which restores the copy if another writer moved on and then adds the copy to currentCopies (:408/:421). The lease listener (onV3LeaseTaken, ~`:1249-1253) deletes the copy from currentCopiesand callsreplace()`.

Suspected race: the retake fires while an in-flight replacement is inside ensureV3CopyCurrent, after its restore read the lineage but before it adds the key:

1. the listener's currentCopies.delete runs first (a no-op: the key isn't there yet);
2. its replace() returns at once (replacing is set);
3. the in-flight ensureV3CopyCurrent then adds the key, marking current a copy restored before the retake, which may lack commits another writer landed in between;
4. serve commits on that copy under the new fence and publishes a lineage from it, so the other writer's acknowledged commits are dropped.

If step 4 holds, that's an S1 (acknowledged commits lost) for serve on any replicated datastore. Reproduce it in process: serve's replication loop with a short probe interval, a stalled ensureV3CopyCurrent (an existing test seam, or a hook in the restore), and a second checkout committing and retaking in the window. If it holds and #3180 hasn't merged yet, say so on #3534 at once; otherwise fix it here first.

The other bugs

1. A retake during serve's in-flight sidecar replacement wedges serve (V3-54, S3 once 4E ships). replace() is single-flight and silently drops a request made while one is running. The listener deletes the copy from currentCopies and calls the no-op replace(). Nothing re-adds the copy, so assertWriterCurrent (:293-302) throws WriterFencedError(... "copy-restoring") on every commit. The probe (:1254-1260) only replaces a dead sidecar, so serve refuses every write until restarted. Reproduced on main (SWAMP_V3_REPLICA): 30 s after the retake, every commit is still refused; control (replacement finished before the retake) passes. Trigger: serve with replication, a sidecar crash, and a lease loss during the 1.5–15 s replacement.

2. "The next write replicates them" can be false (V3-40, S3 once 4E ships). When the closing sync fails, V3ReplicationError (src/domain/datastore_v3/litestream_replication.ts:66-77) exits 75 saying the changes are "kept in commits.db, and the next write command replicates them". That holds only if this checkout writes next. If another checkout takes the lease and publishes first, this checkout's next command sees its copy as foreign (published_lineage.ts:192-232) and makeV3CopyCurrent (v3_copy.ts) restores over it, logging only at info. Reproduced: A's write exits 75 "saved locally"; B writes; A writes again; a reader sees 4 commits, not 5. A's change is in no checkout, and B reused its version number. datastore-litestream.md:1110-1111 accepts the drop; :768-770 and the message promise otherwise. Trigger: a replica outage on one machine while a teammate writes first.

3. The replicator can hand out a released sidecar (V3-55, S4, latent). LitestreamReplicator.acquire (litestream_replicator.ts:219-244) awaits entry.handle.probe(), then does held.refs++ without re-checking #held. #releaseShared (:247-252) closes the handle at 0 refs, so an acquire that overlaps the last release gets a handle to a closed sidecar, and use.lock is free although a caller holds a view. Reproduced with a forced overlap. No current caller overlaps them, but serve's replace/probe and command holds are the paths that could.

4. A non-replicating writer commits to the shared commits.db (V3-56, S4). Reproduced on main: a process without the replica setting writes to the datastore's shared commits.db, which replicating checkouts never read; a replicating reader sees 1 commit, not 2. PR #3180 makes a bucket datastore always replicate (v3BucketBindingAt in v3CommitsDbPath), so this is gone on buckets. It remains for the filesystem test setting. Nothing in the datastore records that replication is on.

What to do

1. The V3-54 neighbour (above): reproduce, then fix. The fix for (1) should cover it: after the in-flight replacement's ensureV3CopyCurrent, re-check that the fence it restored under is still the current fence before adding the copy to currentCopies; if a retake happened meanwhile, restore again.

2. Serve wedge (1): a pending-retake flag. A replace() requested while one runs sets it, and the in-flight replacement runs again after it settles; or the replacement re-checks the fence and currentCopies before finishing. Also let the probe replace when the copy isn't current, not only when the sidecar is dead.

3. Unconfirmed commits (2): pick one, say which:

   • (Recommended) keep the design's drop, but make it visible: change the message to say the changes replicate if this checkout writes next and are lost if another checkout writes first; and have the restore that drops commits past this copy's last confirmed head warn (not info), naming how many commits it dropped;
   • or refuse the restore while the copy has unconfirmed commits (needs a recovery path for the user; bigger).

   Update datastore-litestream.md to match.

4. Replicator race (3): increment refs before the await, or re-check that #held still maps to the entry after it and retry otherwise.

5. Replication as a datastore property (4): on filesystem, record in the datastore (format marker or _control/) that replication is on once a lineage exists, and refuse a writer that doesn't replicate. If you decide the filesystem setting stays test-only and this isn't worth it, say so here and add a test that pins the bucket behaviour instead.

Tests

Each must fail on main (or on #3180's branch where the code moved) first. Use the pinned Litestream as the existing replication tests do.

• V3-54 neighbour: a second checkout's commit landed between the replacement's restore and the retake is present in serve's copy after serve's next commit, and in a fresh restore.
• Serve wedge: retake during an in-flight replacement; serve commits again once the restore lands.
• Unconfirmed commits: A's sync fails (exit 75), B writes, A writes; the message and the warning say what happened (or, with the refusal option, A is refused).
• Replicator race: overlapping acquire and last release keep use.lock shared and the handle open.
• Non-replicating writer: refused once replication is on (filesystem), or the bucket pin.
• On buckets: at least the V3-54 neighbour and serve-wedge tests also run against ministack (S3) with PR #3180's opt-in variable, as integration/datastore_v3_bucket_test.ts does.
• swamp-uat passes unchanged.

Severity: possibly S1 (the neighbour, unverified); otherwise S3/S4 once 4E ships: serve refusing writes until restarted, and a lost change behind a message that says it's kept.

Blocked by: swamp-club#3534 (PR #3180) for the bucket tests; the neighbour check can start now on its branch. Must land before swamp-uat's s3-v3/gcs-v3 suites (Phase 4 H) and before v3 on buckets is announced.