#3916 v3: the compaction policy bounds how much history a fresh clone replays (retainMs, minCommits, retainCommits) and the base load skips files the tail removes

Background

swamp's datastore rework (tracking: swamp-club#2865) finished solo mode on 2026-10-25. swamp-club#3803 made a v3 first clone and restore 3.0–3.6x faster. On the Phase 4 dogfood's repos they're still 2.3–3.2x v2 on emulators, against swamp-club#3418's target of 1.5x, and the gap grows with history: a fresh clone replays every commit compaction has kept.

#3803's compaction experiment showed compaction is the lever. Copies of the dogfood datastores, compacted to keep only the newest 500 commits, cloned at mostly 1.6–1.8x v2 on GCS and about 2.0–2.6x on S3, once the base load stopped syncing every file (that fix shipped in #3803). But today's policy almost never compacts:

// src/domain/datastore_v3/compaction_policy.ts
export const DEFAULT_COMPACTION_POLICY: CompactionPolicy = {
  minCommits: 10_000,
  retainCommits: 10_000,
  retainMs: 7 * 24 * 60 * 60 * 1000,
};

So a fresh clone always replays at least the last 7 days and the newest 10,000 commits. A team's busy datastore would replay tens of thousands. Team mode (Phase 5) brings exactly those repos, so this is part of Phase 5, and it blocks both the team-mode announcement and the Phase 7 default flip.

What to settle and change

1. retainMs, the 7 days. This is the binding limit for a busy repo. compactionBound keeps the period equal to the outbox retry bound (OUTBOX_RETRY_BOUND_MS), "so a commit a writer may still retry … stays a commit". But compacted_txns already keeps each compacted commit's txnId for that bound plus a day (COMPACTED_TXN_RETENTION_MS), so a retry inside the bound never finds its txnId missing. Settle whether a retry can land safely against folded history, and so whether retainMs can shrink or go. Prove it with the outbox retry tests, including a retry that arrives after its commit was folded.

2. minCommits and retainCommits: nothing folds until 10,000 commits are eligible, and 10,000 are always kept. Pick values from measurements (step 4). The trade-off: a smaller retention means a checkout that's behind loads the base more often.

3. Net the tail into the base load. In the experiment, the 500 commits replayed after the base removed about 4,700 files the base had just written. Skip base files the tail removes or replaces, using the same bounded first pass #3803's net replay uses (only from an empty checkout; the replay's own decideFile).

4. Measure, against today's policy, on the dogfood repos (~/dogfood-3756; #3803's scripts and data in ~/dogfood-3756/3803-data/):

   • a fresh clone and a restore;
   • a checkout that's a day, a week and a month behind;
   • one quiet bench.sh round, so writes, reads and workflow steps don't regress (compaction runs on the writer).

   Interleave the runs, use an A/A twin, and report against 1.5x and 2x v2.

Fold in two notes from #3803's final review:

• the duplicated pin in integration/datastore_v3_rules_test.ts;
• the fixed 200 ms sleep in a negative test (AGENTS.md: poll with waitFor, never sleep).

Done when

• The policy decision is recorded in design/enablers/datastore-litestream.md or datastores.md, with the outbox-retry proof.
• Tests fail on main first for any change in retry or compaction behaviour.
• The tail is netted into the base load, and a property test holds it to a full replay.
• Fresh clone and restore on S3 and GCS, measured against 1.5x and 2x v2, with the lagging-checkout cases and no bench regression.
• The swamp-uat datastore suites pass unchanged.

Blocked by: nothing (#3803 has shipped).