Relationships
#4784 Team mode: a member's copy follows serve's compaction by installing serve's base per partition (small partitions whole, hot ones by difference), so copies stay bounded without a local rebuild
Opened by stack72 · 11/16/2026· Shipped 11/16/2026
This implements the caught-up-copy half of "Base install by partition (a copy follows serve's compaction; catch-up by difference)", in the section "v3 team-mode copies: following serve's compaction, catch-up by difference and bounded recovery" of design/enablers/datastores.md (agreed by the datastore-rework architecture review in swamp-club#4783 and merged in PR 3807). That section is the source of truth. This issue states the scope and the gates. It closes the interim debt #4702 slice 1 left behind. Since that change, a member's serve-mode copy never compacts (the F48 fix), so its commits.db grows without bound: about 54 MB after 44 h on member-1 (measured). That was accepted for weeks, not for a release.
Design (agreed)
What R does: when a copy holds at least 2,500 commits below serve's compaction point, it installs serve's own base for each partition where serve's compacted seq is past the copy's. It drops the copy's commits at or below that seq. There is no local fold and no local partition rebuild. The result equals a full base load at serve's point by construction.
Partition sizes decide how (measured on the v3-13 log): 4,309 partitions; base records p50 4, max 78,735; the top three hold 78 % of files. The hottest takes 17 % of commits and changes about 750 distinct keys an hour (measured), so a difference is tens of ms (inferred).
- ≤ 1,000 base rows: installed whole.
- Above 1,000: installed by difference from serve, writing only the rows that changed since the copy's last install from serve. A hot partition is the one serve's folds keep advancing, so a whole install of it on each fold would be F48's O(size) shape again.
This is the same primitive the catch-up-by-difference issue (4(b)) builds on.
Serve API (read-only, behind the same per-caller filter as GET base):
- A base index: each partition the caller may read, with its compacted seq at the current point and the other partitions it shares a file key or a lexical conflict with.
GET basefiltered to a list of partitions, andGET base?partition=P&since=sfor a difference: P's rows changed aftersand its removals afters, or "gone" ifsis below P's prune horizon.
Serve's commit store (a schema change):
changed_seqonbase_records,base_versions,base_catalogandbase_forwards, written by the fold;- a
base_removedtable for rows a fold deletes; - each partition's prune horizon.
base_removedis pruned after the retention age or a size cap. - The schema bump makes an older swamp refuse the store as newer.
- No per-row backfill. Existing rows keep a NULL
changed_seq. The migration writes one difference horizon per partition, at its compacted seq (O(partitions)). Asincebelow the horizon answers "gone", so the copy takes a whole install. - A re-upgrade after a rollback resets every horizon, so a copy never gets a wrong difference.
The install on the copy: one transaction per batch of partitions, under the copy's writer lock. For each partition P:
Replace P's rows in
base_records,base_versions,base_catalog,base_forwards,compacted_partitionsandcompacted_txnswith serve's.- Forwards are partition-local:
renamed_tois a bare name resolved at read time in the same model's catalog.
- Forwards are partition-local:
Chunk counts:
- subtract P's
chunk_partitionscounts fromchunk_refs; - recount P from serve's base records plus P's kept commits;
- release any chunk that reaches zero, as a fold does.
- If
chunk_refsisn't the sum ofchunk_partitionsfor P's chunks, refuse the install and fall back to the full base load.
- subtract P's
File ownership (
base_files), uncontested keys: write serve's ownership with the mark rebased below P's lowest kept local cursor, or at the copy's current cursor if none are kept. A copy's cursors aren't serve's: it stores its own landed commits out of feed order.Contested keys: none exist on the v3-13 log (0 of 182,785 base keys, 0 file/directory conflicts). There's no contested path. If serve's index ever reports one, the batch goes to today's full base load, as a guard.
By difference (partitions above 1,000 rows, whose base came from serve):
- upsert the changed rows and delete the removed ones;
- adjust chunk counts per changed or removed row;
- rebase marks for the changed keys;
- drop the commits at or below the new seq;
- record the new serve seq.
compacted_partitionsrecords whether each partition's base came from serve and at which seq. A partition compacted locally before #4702 slice 1 is installed whole once.Never drop a commit still in the copy's outbox. Serve's
compacted_txnsare installed, so a retry of a folded commit is still a duplicate. An outbox entry below serve's compacted seq is divergence: refuse, and leave it to the existing divergence path.
The read model: unchanged. It has already applied every commit of the reinstalled partitions. The copy's global compaction.point doesn't move, so behind_compaction isn't triggered.
When it runs:
- Inside a command, after its output is flushed and before it returns, never at exit.
- ≤ 1 s per command, resumable between partition batches.
- Skipped when another process holds the copy's writer lock.
- A hot partition that must be installed whole (first install, or past the prune horizon) never runs at a command's tail. It runs in the first-command batch of catch-up by difference (4(b)), or in R's own bounded batch if 4(b) hasn't landed yet. The 1 s tail budget always holds.
Not in scope:
- Serve's own follower copy. It's left to the takeover design (a physical follower plus a server-side seeded lineage), under which it's physical and doesn't need R. If that design slips past R's release, a background local fold on the follower is a separate stopgap.
- 4(b) (catch-up by difference) is a separate issue built on this primitive.
Observability: a fold span on serve and on the copy, recording the partitions folded, the commits folded and the time taken. No fold-level span exists today.
Compatibility (rolling team upgrade)
Serve's store and member copies share one commits.db schema.
- A new member and an old serve: serve names a new datastore feature,
base-difference, inx-swamp-datastore-features. Without it the member keeps slice-1 behaviour (no R; a behind copy takes a full base load) and never fails. If the feature disappears mid-run, it stops between install batches. - An old member and a new serve: unaffected. The endpoints are additive, and existing responses gain no fields.
- A new serve's store opened by an older swamp: refused with
datastore_v3_schema_newer("upgrade swamp … Nothing was changed"). That's the existing downgrade barrier.- Rollback is forward: reinstall the newer swamp.
- A true rollback restores serve's
commits.dbfrom Litestream at a point before the upgrade, and loses every write since. The release notes say so.
- A member copy after a downgrade: the same clean refusal, never treated as corruption. The operator either upgrades again or removes only
commits.dband its-waland-shm, keepingcommits-outbox.db. The older swamp then takes a full base load and flushes the outbox. - A follower serve older than the leader refuses the leader's newer lineage. Upgrade followers before the leader; the release notes say so.
Targets
- A member copy's
commits.dbstays bounded over a 44 h dogfood run: about the size of serve's retained log plus the base, not growing with history. - Users see at most 1 s added at a command's tail, only while R has work. An active member has about 100-160 partitions an hour under the measured load (inferred: about 0.1-0.3 s an hour).
Gates
- A differential property test: after R, the copy's projected content (catalog rows, live versions, files) equals a full base load at the same point. It covers:
- partitions with holes (the follower store);
- a partition the caller may read only in part (config partitions);
- outbox entries pending;
- a contested key (synthetic), routed to the full load as a guard;
- a difference across a pruned horizon, falling back to a whole install;
- the chunk-count sum invariant, before and after every install.
- Fixtures include data written by older swamps, including copies that compacted at exit before #4702 slice 1, and copies whose
chunk_partitionscame from the schema-8 backfill. - The VM gate, against the release and main baselines:
- the copy stays bounded over 44 h;
- quick.sh shows 0 failed recoveries, 0 lost writes and 0 diverged copies;
- while R is folding, K2a and the two-step workflow are no worse than the baselines at p50 and p95;
- no command exit over 1 s.
- Compatibility cases 1-4 have tests:
- a mock serve without
base-difference(no R, full load, no error, and the feature dropping between batches); - the old-client contract (no new fields);
- a schema-N store refused by the N-1 migrator, unchanged;
- a downgraded follower-mode copy refused cleanly;
- the outbox surviving the removal of
commits.db; - re-upgrade after a rollback resetting the horizons.
- a mock serve without
- A hot partition of about 80 k records, folded repeatedly by difference:
- every install stays under the 1 s budget;
- no exit goes over 1 s;
- the copy's rows for that partition equal serve's after each fold.
Process controls
- State scope at every handoff: each status report and the PR description say what this issue closes (R only) and what remains (4(b), 4(c), 4(a), 4(d), 7).
- Validate the VM bench before a build goes out: the token, the serverUrl, and the copy's shape.
- One verify per batch. Targeted tests while iterating. Reruns only for a load or infrastructure red, and say so.
- Gate on VM evidence against the release and main baselines, never on laptop numbers.
- Coordinate any change touching recovery, restore, compaction or takeover with the datastore-rework review and #4679's owners before code.
- Release notes and the #4325 manual carry two operator facts:
- the upgrade order: followers, then the leader, then members;
- a true rollback means restoring serve's
commits.dbfrom before the upgrade, which loses every write since.
- Docs: the agreed design text moves into
design/enablers/datastores.md("v3 history compaction") in this PR, together with the correction that serve's retention pass folds everything older than 24 h with no count floor.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.