#3532 v3: a data name emptied of every version restarts at version 1, reusing a number for different content (use the partition's highWater; refuse a reused version)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the v3 format every change is a commit in commits.db, applied through per-partition rules (src/domain/datastore_v3/partition.ts). Data retention (swamp-club#3473, branch gc-updates) found that a data name emptied of all its versions restarts at version 1 on the next write, so one version number names different content on different clones.

#3473 fixes v2 with a version floor, and on v3 limits retention to keeping each name's latest version as an interim. The interim covers retention only: swamp data delete <name> (every version) on a v3 datastore empties the name today, so v3 can reuse a version number already, without #3473. This issue fixes it in v3 itself, and then removes #3473's interim.

What follows is the retention agent's own write-up, kept as written.

The problem

Data retention can delete every version of a data name (it applies only when garbageCollection.data is set explicitly). On both engines, the next write then started again at version 1, so a version number was reused for different content. Another clone that had pulled the old version 1 would answer data get <name> --version 1 differently from the clone that re-wrote it.

• v2 allocates max(versions on disk) + 1 (atomicAllocateVersionDir in src/infrastructure/persistence/unified_data_repository.ts).
• v3: the partition keeps a per-name highWater that never decreases (src/domain/datastore_v3/partition.ts). But nothing reads it when a version is allocated, and the partition accepts a data-version commit for a number at or below highWater once that number is no longer live. So v3 restarts at 1 too.

Options considered

1. v2 floor record now, v3 later (chosen).
2. Fix both engines in this PR, by wiring v3 allocation to the partition's highWater and making the partition refuse a reused version. Rejected for now: it reaches into v3 internals and means more review and verification rounds.
3. Never fully empty a name: keep its last version's metadata. Rejected: it leaves an old record behind, which weakens #3473's "nothing older remains" guarantee.

What this PR does

v2: a version floor

• When a name's newest version is deleted (by retention, by swamp data delete <name> --version N, or by swamp data delete <name>), the data repository records that number in a per-model file: data/<type>/<modelId>/.version-floors.json, a map from name to floor.
• Version allocation never goes below the floor: max(max on disk, floor) + 1.
• The file is staged and synced like any other data file, so another clone sees the floor after a pull.
• No name walk sees it, because walkers only count directories.
• Tests:
  • allocateVersion: version numbers never go back after a name's versions are all deleted, red-checked;
  • the #2975 test now expects version 3 instead of a reused 2.

v3: interim behaviour

• raiseVersionFloor is a no-op on v3 (typedStagingActive): a plain file isn't one of the changes a v3 unit commits.
• So on v3, retention keeps each name's latest version instead of emptying the name (keepLatestVersion):
  • wired in createDataGcDeps, createDataGcDepsFrom and createRetentionCollectors when resolverEngine(datastoreResolver) === "v3";
  • doctor datastores --retention counts that version as kept by design, not overdue (retentionScanSourcesFor sets keepLatestVersion).
• Consequence: on v3 the strict guarantee is weaker. One version per name (the latest) survives even when it is past the cutoff.

Follow-up work your agent needs to add (datastore-rework, v3)

1. Wire the partition's highWater into v3 version allocation.
   • A v3 write must allocate above max(live versions, highWater).
   • The data repository can't see partition state today: CommitLogEngine (src/infrastructure/persistence/repo_unit_of_work.ts) exposes only openUnit, hasOpenUnit and runRoot.
   • Something has to carry a per-name floor from the partition or read model to atomicAllocateVersionDir, or v3 allocation must move to the commit path.
2. Make the partition refuse a reused version.
   • In Partition.apply, data-version should reject version <= highWater unless it is an append of the same version.
   • Choose the refusal code, for example a version-reused alongside version-live.
   • Make sure replays, compaction and outbox retries still pass (they re-record live versions through the append path).
3. Then remove the v3 interim:
   • drop keepLatestVersion from the three wiring sites, the doctor sources and the pinned engine-reader entries in integration/datastore_write_seams_rules_test.ts (each is commented with swamp-club#3473);
   • update design/surfaces/repo.md ("Version numbers never go back") and design/primitives/data.md.
4. Tests to add with it:
   • a v3 two-checkout test: reap a name fully, write it again from a second checkout, and assert the new version is above the old high-water mark;
   • a partition test for the reuse refusal.
5. Optional: decide whether v2 should eventually move to the same model when v2 is retired. The .version-floors.json files can be ignored or cleaned up then.

Related open items from the same review (not part of this decision)

• A load bench for automatic retention on a large backlog:
  • about 100k runs and about 140k outputs;
  • measure the first pass's time, how many passes catch-up takes, and serve write latency during a pass;
  • run and output batches rescan from the start each batch.
• The CEL data.latest() "reaped" hint is not implemented.

Added by review

• This is a bug on v3 today, through swamp data delete <name>. Add a test for that path too, not only retention's.
• Where to allocate. On v3, a single writer holds the lease, so allocating from committed state at commit time can't race. A checkout's read model can be behind another checkout's commits until it catches up, so don't allocate from the materialised files alone; use the partition's highWater from the commit store, which is current under the lease. Say which seam you chose (staging in the unit, or allocation on the commit path) and why.
• The partition refusal has to survive compaction and restore. highWater must survive into the compacted base (it's in the partition's serialised state) and a D2 restore's truncation. A truncation that drops a version must not lower highWater below versions another checkout may already hold. Test both.
• v2 note, no change: .version-floors.json is one file per model, so two clones that each empty a different name of the same model and push concurrently can lose one floor (last push wins). Retention's hourly claim makes that rare; manual deletes on two clones at once could still hit it. Accept it for v2 and say so in repo.md.

Blocked by: swamp-club#3473 merging (it adds the interim this removes). Should land before Phase 4E puts v3 on buckets, where several machines share one datastore.