Relationships
#4980 Team mode: a booting serve blocks its event loop for 21-51 s on a large datastore, longer than the writer lease's 25 s lapse, so a standby can take over mid-boot
Opened by stack72 · 11/21/2026
Problem
While a team-mode serve boots on a large datastore, its event loop blocks in synchronous work for longer than the writer lease's lapse window. The lease's renewal heartbeat stops for 21–51 s, while the lease lapses 25 s after the last renewal (15 s TTL + 10 s skew). In a two-serve team, a standby can then read the booting leader's lease as lapsed and take a new fence mid-boot.
Evidence (M, dogfood copybench B on the 558 MB v3-13 datastore, main b21dc9b8 and f877b505)
Heartbeat gaps (between consecutive heartbeat putObject spans) during boot:
| Synchronous phase (attribution inferred from overlap) | Cold | Warm |
|---|---|---|
| Before recovery (a near-full read of commits.db, under investigation in the cold-boot issue) | 39–51 s | none |
History tail + reproject (#reprojectNames and the reconcile run with no await) |
25–29 s | 21 s |
| Scan + history head | 11–15 s (some reps) | none |
Unattributed, after /ready |
not seen | 31 s |
- The older main f877b505 also shows boot gaps over 25 s (36.5 s and 41.6 s, during recovery and the run-index repair window). So this predates the recent shorter lease, which only narrowed the margin (lapse 40 s → 25 s).
- Steady state is fine: three 2–4 h soaks show a maximum heartbeat gap of 14.1 s and none over 20 s, including 26–28 s serve folds, which yield.
- The run-index repair (7–19 s) yields, so it doesn't stall the heartbeat.
Not yet verified
- That a mid-boot takeover is safe. Fencing should refuse the old writer's appends, so the expected effect is an extra takeover and disruption, not data loss. The copybench has no peer serve, and quick.sh's rounds don't cover a cold boot at this size.
Scope (timeboxed, about one day)
Verify the hazard: a two-serve test (in-process or integration) where the leader's boot blocks the loop past the lapse. Show what the standby does, and that no write lands twice or is lost.
Fix the starvation, choosing the smaller of:
- (a) lease renewal that doesn't depend on the main event loop (for example a Worker that renews from its own loop, with the same fence and confirm semantics);
- (b) yielding inside the long synchronous boot phases (reproject, reconcile, the open-path reads), bounded so the heartbeat never misses its interval.
The confirm invariant (the holder stops at its monotonic asked + TTL) must hold in either case. Any change to fencing or the confirm logic is escalated before it's built.
Spans around every boot phase longer than 1 s, so no stall is unattributed.
One full verify, then one VM run: copybench B (cold and warm) with the heartbeat gap distribution, plus quick.sh for safety.
At most one fix round. Review suggestions are listed in the PR, not folded.
Stop rule: if the fix needs a new mechanism beyond (a) or (b), stop and report.
Done when
- No heartbeat gap over 10 s during a cold or warm boot at 558 MB.
- The mid-boot takeover test shows safety: no duplicate or lost writes.
- quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.
Added: a concrete one-off case (from the cold-boot issue's trace, 2026-11-20)
The first open of a commits.db at schema 13 by a binary past dce51ecff runs the 13 → 14 migration. Its ADD COLUMN on five STRICT base tables makes SQLite check each table's existing rows, synchronously on the main thread.
- Measured: on the 558 MB dogfood store, cold, that's 123,628 random 4 KB reads (483 MB, 48.8 s of syscall time) with no heartbeat.
- Measured locally: 117 ms warm on a 534 MB STRICT table, against 1 ms non-STRICT.
So a serve upgraded and restarted cold stalls well past the 25 s lapse, once. This issue's fix should cover it: renewal off the main loop covers it directly, while yielding doesn't, since it's a single SQLite statement. Product options recorded, not chosen:
- keep changed_seq in a side table rather than as a column on the STRICT tables;
- run the migration before taking the writer lease.
Open
No activity in this phase yet.
Sign in to post a ripple.