Relationships
#4432 Team mode: serve confirms replication before a graceful stop, confirms each commit's own txid, and accepts tokens once ready (F22, F19, F30 from #4374)
Opened by stack72 · 11/5/2026· Shipped 11/5/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), runs 2 and 3, release 20261105.030003.0 (contains #4387 and #4401). Code references are at 363a1dad. This is one of the two pieces of work that gate dogfood run 4 (target: shipped by Mon 9 Nov).
Three problems in how a v3 serve starts, stops and confirms replication. F22 is the only possible data-safety finding left from the dogfood.
1. F22: a graceful stop shuts down replication before confirming its last commit
- Symptom: every
systemctl stopof the leader logs "This command's changes are saved locally, but swamp could not confirm that they reached the datastore's replica: the Litestream sidecar failed: no control socket at …/ls.sock … if another checkout writes to the datastore first, they are lost". The process exits non-zero (systemdfailed). Stopping a follower is clean. - Evidence: the leader's stops at 01:57 UTC (run 2), and 08:58, 12:2x and 15:49 UTC (run 3), plus the stops in the takeover checks. The journal shows
Shutdown: stopping "v3 replication" … stopped in 1 ms, then the error about 5 s later. - Likely cause: in
src/cli/commands/serve.ts(~8113–8120), the"v3 replication"shutdown stage (replication.stop(…)) runs before the closing work that confirms the final commit (closeV3Datastores:POST /sync, thenreplicated_txid >= txid). That confirm then finds no sidecar. - Fix direction: confirm the closing commits (sync, check
replicated_txid, release the writer lease) before stopping the sidecar, or havereplication.stopdo the closing confirm itself. A graceful stop must not exit with unconfirmed commits. - Why no acknowledged write was lost: members are only acknowledged after the confirm, so the unconfirmed commits are serve's own closing commits. A follower taking over from the replica would still miss them.
2. F19: Litestream reports "synced" one transaction short, and serve fails the write
- Symptom:
Litestream answered {"status":"synced", … "txid":N, "replicated_txid":N-1}; serve logs "Error withheld from client (V3ReplicationError)…" and the request fails. In run 3 it reached a member: a vault read through serve returned exit 75datastore_v3_replication_unconfirmed. - Frequency: 11 times in run 3's set 1 alone (10 on one node, 1 on the other), more than run 2's 5.
- The check:
isReplicatedinsrc/domain/datastore_v3/litestream_replication.ts(~132–138) requiresremote >= local. - Hypotheses: the confirm reads a
txidthat includes a write made after the sync was requested (a checkpoint, compaction or marker write), or Litestream'sreplicated_txidlags by one LTX by design. - Fix direction: find which write lands between the commit and the confirm, and confirm the commit's own txid rather than the file's latest. Only if the commit's txid really is covered, retry the sync once on
remote == local - 1before failing. Do not weaken the check to accept a commit that is not replicated.
3. F30 (candidate): a node refuses a valid token just after it restarts
- Symptom: about 20 s after a serve node restarts, with
/readyanswering 200, a valid probe token is refused on that node while the other node accepts it. - Evidence: run 3, set 1
t03and set 2t02, one on each node. - Check first whether #4401's token-secret cache (
09c042bd0) causes it, for example a cache filled before the node has caught up, or a cached miss. If so, it is a regression from that change. Either way,/readymust not answer 200 before the node accepts the tokens the other node accepts.
Also look at (does not block)
- F24: in a SIGSTOP pause round, one member's writes stalled for 131–199 s while the other member resumed in about 5 s (3 times across runs 2–3). Record what you find while in this code.
Must not regress
- No acknowledged write lost; 0
serve_upstream_diverged. - Failover write pause 4–7 s; follower restore about 10 s, ready in about 60 s.
- Revocation refused on the next request on both nodes, and after a takeover.
Done when (dogfood run 4 checks)
f22_checkafter every graceful stop of the leader shows 0 "could not confirm" lines, and the unit exits cleanly.- The serve-log scan shows 0
V3ReplicationError … replicated_txidlines across a set. t02andt03accept a valid token on a node right after it reports ready.
In-repo tests: a shutdown-order test that fails on the current stage order; a red-checked F19 test reproducing the off-by-one confirm (a fake Litestream answering replicated_txid N-1 for a commit that is covered, and one that is not); an F30 test that a restarted node accepts a valid token as soon as it reports ready.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.