Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4783 Team mode item-4 design note: a copy follows serve's compaction, never falls behind it, catches up by difference, recovers through an undo log, and bounds a full rebuild

Opened by stack72 · 11/16/2026· Shipped 11/16/2026

Continues swamp-club#4702 (closed after slice 1, PR #3805, outcome not met). The deliverable here is a design note to the datastore-rework architecture review, agreed before any code. Each mechanism below gets its own implementation issue only once its design is agreed. The no-new-mechanism work is in the companion slice-2 issue.

Every mechanism must keep #4581's and #4677's correctness argument (targeted equals full, row for row) and the trust gate. Each one later gets its own differential property test against a full rebuild, and its own VM gate.

What the note must decide

# Mechanism Target it moves Now (measured on the dogfood VMs)
R A copy follows serve's compaction (required, first). Fold a member's serve-mode copy only through serve's compaction point, using serve's base, with no local whole-partition rebuild. Decide how a copy learns serve's point, and whether it's bounded per command or done in the background Member copies stay bounded #4702 slice 1 stopped copies compacting at exit (that's the F48 fix), so a copy's commits.db now grows without bound. Accepted for weeks, not for a release
4(a) Never fall behind compaction in normal use: compaction waits for known copies up to a size/age ceiling, or the cheap option, retention raised to about 7 days with a size cap. Cost the cheap option first. With the architecture review's takeover design (~/Desktop/takeover-design-decision.md: a physical follower plus a server-side seeded lineage, no full restore on takeover), a larger retained log costs a member's catch-up and a cold restore, but not takeover A member back after a long absence usable < 30 s Retention is 24 h of age, so a member away more than 24 h (any Monday) pays a full base load: minutes, growing with datastore size
4(b) Catch up by difference: when a copy is behind anyway, apply the change between its last base and the new base (and the commits after it), touching only what differs Same as 4(a) Same as 4(a)
4(c) Recovery cost independent of a name's history: the catalog undo log. Writes record pre-images per unit; recovery restores the dead writer's names in reverse and applies only the commits after its start. A fast path with today's reproject as the fallback, behind the same trust gate Member recovery p50 < 5 s, max < 30 s About 38 s on 17 names (B on ef94032c1): catalog write 15.5-21.9 s re-applying whole action histories, history read about 14 s, scan about 7 s, compare floor about 2.8 s
4(d) A bounded full rebuild (first clone, a different principal): parallel, streamed, writing eagerly only the files a reader needs; batch the replay guard's chunk calls (#3893 ordering). Joining is an explicit target: a fresh member clone plus its first data list Full rebuild < 60 s, after the login reset as well as on a first clone. Joining (fresh clone plus first data list) ≤ 1.5× v2 on the same VMs About 1,988 s on serve-a's copy (release); 536 s on a member in run 7. Joining: 37.5 s near and 40.7 s far against v2's 2.95 s and 3.61 s (12×, measured 2026-11-13), growing with history (4.8 s fresh to 31-41 s after a few hours)
7 A moved filesystem datastore with a replica: re-key its local copy on something move-stable (the path relative to the checkout when inside it), with a migration, and reclaim the orphaned directory No full restore on a move v3CommitsDbPath hashes the absolute path, so a move restores in full, starts a new lineage and leaves the old copy behind. Not on team mode's path (bucket and member copies are unaffected)

Carried findings the note must account for

  • The history read (about 14 s) decodes and parses every row of a touched partition; a cut needs commits indexed by name.
  • An old view record (with no repoId) can't tell a deleted external datastore from a moved one. A real guard needs the read model to record its datastore's identity (lineage or store id).
  • The view record is rewritten without the writer lease on the fast path. It only matters with two views of one checkout open at once (unsupported); a conditional write or a write under the lease would close it.

All are listed with numbers in ~/Desktop/4702-item4-findings.md, and the overall plan is in ~/Desktop/4702-remaining-work-plan.md.

Process controls (from #4702 slice 1)

  • Design before code: no 4(x) or R implementation until the note is agreed. Each agreed mechanism is filed as its own implementation issue, with its targets and gate.
  • State scope at every handoff: which mechanisms are designed and agreed, and which remain.
  • Fixtures include data written by older swamps, and VM benches are validated (token, serverUrl, move shape) before a build goes out.
  • One verify per batch. Gate each mechanism on VM evidence against the release and main baselines, never laptop numbers.
  • Coordination: recovery, restore, compaction and takeover are shared with #4679's work and the architecture review's takeover design, so agree any change there first.

Done when

The design note is written, reviewed and agreed by the architecture review and the user, and one implementation issue is filed per agreed mechanism (R first).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 5:30:02 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 4:22:25 AM

Sign in to post a ripple.