Relationships
#4824 Team mode: faster takeover, piece 1: a shorter writer-lease TTL (then the follower's physical copy)
Opened by stack72 · 11/16/2026· Shipped 11/18/2026
Problem
When the leader serve dies, the follower takes over. On a real-size datastore (558 MB commits.db, the v3-13 dogfood datastore) takeover to the first acknowledged write is about 86 s (I), against a 60 s target, and it grows with database size. The parts:
- Waiting for the old lease to lapse: the writer lease TTL (30 s) plus the clock-skew allowance (10 s); about 37 s observed (I).
- A full restore from the bucket: 20.8 s at 558 MB for a fresh lineage (M), because the follower's copy is never at the head. It was "behind" in 96 of 96 dogfood takeovers (M).
- A full snapshot upload for the new Litestream lineage: 27.8 s at 558 MB (M).
The earlier attempt at part 3 (seeding the new lineage by server-side copy) stopped: on Litestream 0.5.17 the new lineage's first upload is still a full database image (320,085,046 bytes at 558 MB, M). That finding is recorded below, and part 3 is out of scope here.
Scope: two independent pieces, shipped separately
Piece 1 (first): a shorter writer-lease TTL. The writer lease gets its own TTL constant, cut from 30 s to a value proposed from measured renewal latency (about 15 s, I). Saving: about 15 s, so takeover goes from about 86 s to about 71 s (I).
- The clock-skew allowance stays at 10 s. A leader keeps serving while its last renewal is younger than the TTL (on its own monotonic clock), and a taker waits for the heartbeat plus the TTL plus the skew (on its own wall clock). That keeps a frozen or cut-off old leader from granting model locks or serving stale reads after a takeover, but only while the real clock offset between nodes stays under the skew. No measured offset bound exists, so the skew does not change.
- DEFAULT_LOCK_TTL_MS stays 30 s. It is shared with v2 file locks, and serve's model locks keep their own 30 s value.
- Committed data stays fence-safe at any TTL. Before acknowledging a write, the leader lists the lease for a higher fence. If that list misses the new fence, the write's head was published before the new fence existed, and the new leader reads it after taking the fence.
Piece 2 (after piece 1 ships): the follower's physical copy (B), which removes the about 21 s restore. Takeover would then be about 53 s (I). Piece 2 carries the same full-image caveat as part 3, so it is planned after piece 1.
Steps for piece 1
- Before code: trace every use of the lease TTL and skew, and say whether a shorter value could let two writers land or only cause an earlier takeover (done: summary above). Then measure the cost from existing quick.sh and soak span data, with no VM time: the latency tail of
swamp.datastore.v3.writer_lease.check(the same lease list a renewal runs) andswamp.datastore.v3.lineage.publish_head(small puts, like the heartbeat), lease-loss log lines, and the inter-node clock offsets. Propose the TTL from the tail. Renewal runs every TTL/3, so a renewal needing more than about two thirds of the TTL leaves a healthy leader unconfirmed. - Build: the new writer-lease TTL constant and its uses (the renewal interval, expiresAt, waits), with tests showing the confirm invariant (the holder stops at asked+TTL, the taker waits renewedAt+TTL+skew) at the new value.
- One full verify, then one VM run on the normal fresh datastore: quick.sh failover rounds against the baseline, and a soak with idle stretches counting false takeovers and lease losses outside the rounds. The lease cut does not depend on datastore size; the 558 MB takeover rig is built for piece 2, whose gain does.
- At most one fix round, and only for a miss against the done criteria. Review suggestions are listed in the PR, not folded.
Stop rule: if the VM run misses and one fix round doesn't close it, stop and report.
Done when (piece 1)
- quick.sh takeover p50 and max about 15 s shorter than the baseline, measured;
- no false takeovers and no lease losses outside failover rounds during the soak, including idle stretches;
- quick.sh reliability unchanged: 0 lost, 0 diverged, 0 failed recoveries.
The 60 s target needs piece 2 as well.
Finding kept from the stopped seeded-lineage attempt (M)
- On Litestream 0.5.17, a sidecar started on a restored database writes a full database image as its first L0, even onto a lineage seeded with the old lineage's files. At 558 MB that was 320,085,046 bytes, landing 15–26 s after the sidecar started.
- The cause is in Litestream's sync verification: the local position's WAL salts must match the current WAL. A restored database has a new WAL.
- Litestream 0.5.17 LTX files carry no checksums.
restore -fcannot take-txid.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 11/17/2026, 1:45:40 PM
Closed as stopped by the stop rule, outcome not met, nothing built. Step 1 at 558 MB showed the seeded lineage's first L0 is a full database image (320,085,046 bytes), so takeover still pays the full upload. The evidence and the next-round question (making Litestream continue from a restored database's position) are in the issue body. The S3 and GCS copy method, swamp-club#4825, is closed with it. The physical follower copy stays an unfiled draft for next-round planning.
stack72 commented 11/17/2026, 4:08:00 PM
Reopened with a new scope: the takeover problem stands (about 86 s at 558 MB against 60 s). Piece 1 is a shorter writer-lease TTL under its own constant, with the clock-skew allowance unchanged; piece 2, the follower's physical copy, follows separately. The body is updated.
Sign in to post a ripple.