#4702 Team mode: a member's or serve's copy is never slow to become usable (recovery, serve boot, catch-up after idle or compaction, login reset) — cost proportional to change, never to datastore size

Rewritten 2026-11-13 by the datastore-rework architecture review. This replaces the earlier body. Team mode ships only when it is reliable and performant (no worse than v2 on the same machines, plus the targets below). This issue owns one requirement of that bar:

A team member's or a serve's copy of the datastore is never slow to become usable: after an interrupted command, after a kill or a failover, after being idle or away, after a login change. Its cost is proportional to what changed, never to the size or age of the datastore, and nothing slow runs while holding a lock other writers need.

Follows swamp-club#4581 and swamp-club#4677 (both shipped with targets not met). Evidence from team-mode dogfood swamp-club#4374 (run 7, quick.sh runs, copy benchmarks on the dogfood VMs). Coordinates with swamp-club#4679 (failover).

Targets (all measured on the dogfood VMs; laptop numbers don't count)

Hard rule: no operation whose cost grows with the datastore's total size (a full base load, a full rebuild, a whole-checkout walk) may sit on the path of an ordinary command or a takeover. Where one is unavoidable (a new member's very first clone), it must be bounded and measured.

What the evidence says (where the time goes)

1. Targeted recovery still re-derives every touched name from its whole history (member-2 copies, main): reproject 20–31 s, history read 10–19 s, scan 5–11 s. Versions accumulate in real use (data GC is opt-in: autoGc off by default), so names with thousands of versions are normal.

2. Serve boot on the release (copybench B, serve-a's own copy, 14 dirty names): targeted readmodel.recover (interrupted_writer) 503 s: reproject 257 s, scan 45 s over 191,949 files, history 19 s, ≈ 180 s untraced. The serve was then ready. (The bench's second recovery was an artifact of running the copy at another path; see 2b.)

2b. A full rebuild at dogfood size takes 1,988 s (33 min), and moving a checkout triggers one. The bench copy ran at a different path, checkout_view.json records the copy's absolute path, so the boot took view_change → full rebuild: 22,110 commits, 100,848 files written, 7,572 chunks (100 MB, fetch 497 s), remove 193 s, guard 260 s, catalog 282 s, base load 875 s. The same full rebuild is what a login reset to a different principal (F54) and a first clone pay. Moving or renaming a checkout directory must not force it at all.

3. Falling behind compaction forces a full base load (a design risk; not what the copybench boot hit, which was 2b). #catchUpFeedRun / #catchUpPartition in sqlite_read_model.ts call #loadBase whenever the cursor is behind the commit log's compaction point (CommitsCompactedError). #loadBase stages the whole compacted base, writes every live record's files (≈ 190k here; chunks fetched from the bucket) and rebuilds the catalog for every name (≈ 78k rows). DEFAULT_COMPACTION_POLICY keeps only the last 2,500 commits / 24 h, a few hours at dogfood write rates. So a member away overnight, a serve down for a while, or a copy restored from an older lineage pays a cost proportional to the whole datastore. Measured on serve-a's copy: cursor 52,610, compaction point 30,500, so a copy caught up within retention does not hit it; one away longer than the retention does.

4. Read-model writing dominates the cold read: projection 2.1–2.3 s near (≈ 1,064 files for 160 commits, prefetch, bookkeeping); commit apply is now ≈ 0.3 s after #4631.

5. Lock interplay: a catch-up of 751 commits ran under the behavior's lock for 64 s; after a SIGKILL the next write waited 20–25 s for the dead command's lease.

Work, in order

1. First, a correctness bug: swamp-club#4723. With a pinned fast-check seed, targeted recovery leaves data/<model>/<name>/latest at a different version than a full rebuild after a rename onto a deleted name with an interrupted writer (deterministic, fails on main after #4677). Find which side is wrong (the full rebuild points c/latest at version 1 while no c/1/* files exist), fix it with the pinned seed as a red test, and add the shrunk history as a fixed regression case. Fix it under #4723's own lifecycle and PR, not linked into this one, so the correctness fix merges as soon as it is ready instead of waiting for this issue's larger work. Same agent, done first: everything below builds on the targeted-recovery code it touches.
2. A moved checkout must not rebuild: checkout_view.json keys the view on the copy's absolute path, so moving or renaming the checkout forces a full view_change rebuild. Key it on something stable (the datastore and principal), with a red test. Cheap and first.
3. Measure on main (copybench A/B/C on ~/team-old-13 copies, run by you through ~/dogfood/QUEUE.md): member recovery, serve boot and full rebuild, with phase breakdowns.
4. Cheap fixes (no new mechanism): running maximum in catalogActionsForNames; count(*) instead of loading every key into a Set; compare version numbers before parsing metadata; filter history by op keys before Commit.fromJSON; trace the ≈ 180 s untraced part of recovery.
5. Design note to architecture before any code for the new mechanisms below. Each one must keep #4581's and #4677's correctness argument (targeted equals full, row for row) and the trust gate.
   • (a) Never fall behind compaction in normal use: compaction waits for known copies (serve knows its followers' and members' positions) up to a size and age ceiling (for example 7 days); a follower serve applies the leader's commits continuously so it is never behind.
   • (b) When a copy is behind anyway, catch up by difference: apply the change between the copy's last base and the new base (and the commits after it), touching only what differs, instead of rewriting every file.
   • (c) Recovery cost independent of a name's history: the catalog undo log from #4677's handover (pre-images per write, restored in reverse), as a fast path with today's reproject as fallback.
   • (d) A full rebuild that has to run (first clone, a different principal) is bounded: parallel, streamed, with only the files a reader needs written eagerly.
6. Read-model writing on catch-up: cut the projection cost of the cold read (fewer files written eagerly, batched writes, prefetch overlapped).
7. Lock interplay: catch up before taking the lock and apply only the remainder under it; the next command from the same checkout reclaims its own dead predecessor's lease immediately, only when the holder is provably dead on this host (same check as the writer marker) and only for a lease this checkout held; fencing unchanged.

Slicing: agree it with architecture in the plan. Each slice merges on its own evidence.

How to measure, and the merge gate

• No speed number from a laptop counts. Local runs prove correctness; speed comes from the VMs.
• Iterate on VM copies with copybench.sh <build> <A6|A14|B|C> (#4374 session; long-history copies in ~/team-old-13, release vs build, read-only). Queue through ~/dogfood/QUEUE.md on dogfood-ctl; never overlap a serve restart or kill round.
• Merge gate: the PR goes up with the hold label and comes off only after a passing quick.sh run (member-kill and failover rounds) and the copybench scenarios it targets, against the release baseline.

Tests

• Count tests: recovery, catch-up and base work bounded by what changed (files touched, partitions read, rows written), asserted by counts, not wall-clock.
• #4581's and #4677's differential property tests (targeted equals full, row for row) and mutation checks keep passing; each new fast path (difference catch-up, undo log) gets its own differential property test against a full rebuild.
• A compaction-retention test: a copy within the retention window never takes the base-load path.

Must not regress

No failed recovery, no lost acknowledged write, no diverged copy. #4512's stoppable, resumable recovery; #4579's refused-read handling; F43's repair; #4677's bounded walk and trust gate.

Coordination

• swamp-club#4679 (failover) owns serve's takeover restore (v3_repo_context.ts around the double restore), background old-lineage deletion, follower WAL checkpointing and the inFlight=true half of serve_commit_sink, and is writing a design note on taking over from a follower's own copy. Agree any change in those areas with that session first; item 4(a)'s "follower never behind" must be designed together with it.
• The #4374 session runs every VM measurement; ask through the architecture review if you need a new kind.

Evidence to read

4677-architecture-handoff.md, 4677-validation-plan.md, 4677-recover-span-breakdown.py (ask the user for copies); the #4374 reports 4374-quick-release-baseline.md, 4374-quick-main-acd18855.md, 4374-attribution-save-speed-2.md, and the copybench B report when it lands.