Relationships
#4899 Team mode: a serve booting with a cold page cache takes 109-122 s to become ready on a real-size datastore (reads about 1 GB)
Opened by stack72 · 11/19/2026
Problem
A team-mode serve that restarts with a cold page cache (after a host reboot, a new VM or memory pressure) takes about 2 minutes to become ready on a real-size datastore. The goal is 60 s. With a warm cache the same boot takes about 57–78 s.
Evidence (M, dogfood copybench B, 2026-11-16/17)
Bench: a serve booting on a copy of the 44 h v3-13 datastore (558 MB commits.db), with recovery after an interrupted writer (targeted). Both main and the slice-2 build behave the same.
| Boot | Read from disk during boot | Ready | Recover |
|---|---|---|---|
| warm cache (11 boots) | 64–103 MB | 57–78 s | 37–45 s |
| caches dropped (3 boots) | ~1,046–1,060 MB, ~77 k IOs | 109–122 s | 76–90 s |
- Per-phase on warm boots: scan ~7–8 s, history ~15–21 s, reproject ~17–26 s; access → running_runs_found ~14–16 s.
- On cold boots, history and reproject roughly double, and the boot reads about 1 GB, nearly twice the commits.db size.
- Raw data: dogfood-ctl ~/dogfood/results/cb/ (cbB-20261116T230856Z onward), with vmstat and diskstats snapshots at boot start and /ready.
Scope (timeboxed, about one day)
- First, attribute it from the existing runs (no VM time): which phases read the ~1 GB, and why more than the database's size? Is it the whole history read in recovery, the reproject over every action (73,856), a full-table scan, or WAL/page-cache effects? Name the largest share with numbers.
- Then a fix sized from that, with tests and one full verify. Then one VM run: copybench B, 3 cold reps (caches dropped) and 2 warm, main against the build.
- At most one fix round, only for a miss against the done criteria. Review suggestions are listed in the PR, not folded.
Stop rule: if the fix needs a new mechanism (for example a persisted read model so boot skips the reproject), stop and report before building it.
Done when
- Cold-cache boot to /ready ≤ 60 s on the 558 MB datastore (median of 3).
- Warm boot no slower than main.
- quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.
In Progress
Click a lifecycle step above to view its details.
stack72 commented 11/20/2026, 1:02:14 PM
Ruled out: start targeted recovery's replay from the interrupted writer's last change instead of from the compaction point.
The idea was to trust the stored rows of touched names below that point and replay only the commits after it. It's unsound as the code stands:
- There is no bound to start from. The recovery markers (
catalog_dirty) record names only, with no seq. Pins record partition and txn_id only, are written only when a unit ends cleanly (releaseDirty), and a killed writer leaves none. - Writers change older rows outside the read model's cursor.
- Settling a new version flips the latest flags on the previous version's row. Every data write does this.
data.deleteanddata.gcremove rows and files of any version.- Renames and forwards change how the whole name resolves.
- A latest-marker repoint changes no row, which was #4723. Trusting stored state would have reproduced it.
- Making it sound needs a new stored mechanism. Each writer would have to record its starting position per partition and undo records for every row it changes, which the stop rule excludes.
Difference-only writes in the reproject already ship (#4677, #reconcileNames). The remaining cost is replay length, and the work goes on with levers that don't change what recovery trusts, sized from the chain17 measurements.
Sign in to post a ripple.