Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4782 Team mode slice 2: cold-read catch-up writing, catch-up before the lock, reclaiming a dead predecessor's lease, a failed lock-time catch-up goes offline, batched offline flush, and serve boot's running-runs scan

Opened by stack72 · 11/16/2026· Shipped 11/16/2026

Continues swamp-club#4702 (closed after slice 1, PR #3805, outcome not met). This issue holds the work that needs no new mechanism. The new mechanisms (copy follows serve's compaction, never behind compaction, catch-up by difference, the undo log, a bounded full rebuild) go in the companion item-4 design-note issue, and are not in scope here.

Team mode ships only when it is reliable and performant. This issue owns the targets below, all measured on the dogfood VMs (laptop numbers don't count).

Scope (all of it; nothing else)

# Item Target Now (measured)
5 Read-model writing on catch-up (the cold read): fewer files written eagerly, batched writes, prefetch overlapped with apply First command after idle about 2× warm (decided 2026-11-16: the levers here reach about 2.1×; ≤ 1.5× needs lazy file materialization, its own item when picked up) near 2.9-3.3 s vs 0.39-0.43 s warm; far 5.1-5.3 s vs 0.9 s. Projection 2.1-2.3 s near for about 1,064 files over 160 commits
6 Catch up before the lock: catch up before taking a behavior's lock, then apply only the remainder under it Catch-up under a lock: a few seconds at most 64 s once (751 commits, 9,346 files) blocking other writers
6 Reclaim your own dead predecessor's lease: the next command from the same checkout reclaims a lease it held, only when the holder is provably dead on this host (the writer-marker check). Fencing unchanged No wait after a killed command 20-25 s
6b A failed lock-time catch-up during a failover: an offline-allowed member whose bind and lock succeed, but whose catch-up under the lock fails ("fetch failed" on GET changes, 0.7-4.1 s after a frozen leader resumes), exits 75 serve_unreachable instead of going offline. Decide with item 6: catch up before the lock, or let a failed lock-time catch-up on an offline-lease checkout go offline. Add it to quick.sh's G2(2) count 0 such exits 1 in #4679's part-2b run; 3 of about 600 attempts in its re-gate on 8f8619d88
6a Batched offline flush: send queued offline entries through the batch route. That needs a batch send on the flush port, a rebase on the projected state (copy plus earlier built entries), per-entry outcomes, and the restate, floor and lease-lost retries reworked. Keep #4679's lost-write guarantees (renumber on version-live/reused, never drop a rejected entry). #4679's estimate: about 250-300 lines plus about 150 of tests; moderate-to-high risk (it changes the rebase basis) 5 s per started block of 20 inherited entries (quick.sh's inherited-flush line) 1.1-1.5 s per entry; 33 of 230 inherited flushes over target (quick.sh g2v2 on dc31a30a9)
— Serve boot's running-runs scan: findGlobalByStatus("running") walks every workflow's run index over the whole history. Use an index of running runs, or bound the boot scan Serve boot below about 45 s on the dogfood history The scan alone takes 12.9-14.0 s of boot (the swamp.serve.boot phase event running_runs_found; copybench B and B-moved on ef94032c1)
F41 A command's exit after large outputs: teardown cost, no new mechanism A command's exit ≤ 3 s Up to 8.6 s near and 6.0 s far (run 7, m_big 17+64 MiB)
— The A6/A14 bench fix (bench-only): the team-old-13 snapshot's view caller is a since-reissued token, so every arm takes view_change before the entry guard. Rewrite the snapshot view's caller to sha256 of the token in use, or run with the original token A6/A14 measure targeted member recovery on main and on the build Not measurable (aborted twice on ef94032c1)

Out of scope: anything that needs the item-4 design note (the undo log, catch-up by difference, compaction waiting for copies, a bounded full rebuild, copies following serve's compaction).

Process controls (from #4702 slice 1)

  • State scope at every handoff. Each status report and the PR description say which rows above are closed and which remain.
  • Validate the VM bench before the build goes out: the bench copy's view matches the token and serverUrl in use, and any move is the move the test claims. Slice 1 lost rounds to bench setup, not code.
  • Fixtures include data written by older swamps (view records, stored rows). Two slice-1 VM failures came from shapes that only exist on long-lived copies.
  • One verify per batch. Targeted tests while iterating; the full verify only before a VM handoff or the PR; reruns only on a load or infrastructure red, and stated as such.
  • Gate on VM evidence against the release and main baselines: copybench (A6/A14 with the fixed bench, B), a soak window, and quick.sh (member-kill and failover rounds, G2(2) count included) as the merge gate.
  • Coordination: lock and lease changes are agreed with the #4679 owners first. Item 6's lease reclaim must not weaken fencing.

Must not regress

No failed recovery, no lost acknowledged write, no diverged copy. #4512's stoppable, resumable recovery; #4579's refused-read handling; F43's repair; #4677's bounded walk and trust gate; #4679's offline-flush guarantees; #4702 slice 1's moved-checkout and F48 results.

Evidence

#4702's body and its slice-1 PR #3805 (VM gate table). The #4374 reports: copybench B and B-moved on ef94032c1 (cbB-20261115T213156Z, cbB-20261115T211647Z), quick.sh 20261115T214624Z-4702-ef94032c, and #4679's gate reports for 6a and 6b.

Outcome (a9b9f098, measured on the dogfood VMs 2026-11-16)

Decided 2026-11-16: #4782 ships at this state; what remains moves to the next round, filed when picked up.

# Item Target Delivered
5 Cold read about 2× warm 3.35× warm (from 3.9×): tail netting from the fetched pages, pooled inline writes, prefetch overlapped. Remaining levers (hand the parsed commits to the read model, cut the path checks, drop netting if it stops paying) and ≤ 1.5× (lazy file materialization) move to the next round
6 Catch-up under the lock a few seconds Bounded to 128 commits still to project, 3 rounds, then the rest with a warning
6 Dead predecessor's lease no wait after a kill Met: longest lock acquire after a kill 494 ms; kill recovery p50 0.02 s, max 1.16 s
6b Failed lock-time catch-up 0 exit 75 Met: G2 0 / 0; offline only under a live explicit lease, exit 75 otherwise
6a Batched offline flush 5 s per started 20 Inherited flushes over the limit 12 → 2
F41 Exit after large outputs ≤ 3 s Met: 0.04-0.2 s. Big outputs from member-far are 2.5× v2 (84-112 s against 26-46 s), almost all the member→serve upload (about 1.7 MiB/s): open, next round
— Serve boot's running-runs scan below about 45 s Recovery parks run indexes for a fingerprint repair instead of a full rebuild (the 13-16 s was three rebuilds)
— A6/A14 bench measurable Met: valid on every rep, recovery level with main

Safety on every gate: 0 failed recoveries, 0 lost writes, 0 diverged copies; soak exits all under 0.5 s. Merge waits on the BAAB copybench B re-check (the build's slow position-2 rep, unexplained so far).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 10 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 8:06:51 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 4:23:27 AM

Sign in to post a ripple.