#3239 Datastore rework Phase 3: the v3 read model, dormant (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3:

• every change is an explicit commit in a single-writer SQLite commits.db, replicated by Litestream;
• content is stored as immutable, content-addressed chunks;
• the read model is the local files and catalog that model code, CEL .path and queries read. It is a cache, rebuilt from commits.

Phase 3 builds the v3 engine dormant: nothing reaches it until a user opts a new, empty datastore into v3, which is a later issue.

What's on main (1b48d5b4). All of it is dormant and documented in design/enablers/datastores.md under "v3 domain model", "v3 commit store" and "v3 chunk storage":

• The domain model (swamp-club#3233, src/domain/datastore_v3/):
  • Commit (namespace, partition, seq, txnId, ops). An unnamespaced datastore's commits have namespace "".
  • Ops:
    • put writes the record at its RecordKey, a namespace-relative, forward-slash cache path in today's cache layout. Its content is inline (8 KiB or less) or a BlobRef.
    • tombstone removes the record at one key.
    • data-version carries the DataMetadata that feeds the catalog.
    • lease carries a holder and a fence.
  • A data save is: put the raw content, put metadata.yaml, a data-version op, and the latest marker (see the save mapping in the domain-model section).
• The commit store (swamp-club#3236, src/domain/datastore_v3/commit_store.ts and src/infrastructure/datastore_v3/sqlite_commit_store.ts):
  • changesSince(cursor, limit) returns CommitChanges in landing order, with a store-wide, never-reused cursor;
  • readPartition and head;
  • one commits.db per datastore.
• The chunk store (swamp-club#3237, src/domain/datastore_v3/chunk_store.ts): ChunkStore.read(namespace, blobRef, signal) returns a verified stream. The filesystem storage/1 reference implementation is src/infrastructure/persistence/fs_datastore_storage.ts.
• Today's catalog: _catalog.db, through CatalogStore (src/infrastructure/persistence/catalog_store.ts, upsert, upsertNewVersion, recordRename, …).

A decision this issue builds on: on v3 in Phase 3, who writes the local files? The design's end state is that repositories stop writing files and the read model materialises everything from commits. For Phase 3 we deliberately go part of the way:

• Repositories keep writing local files exactly as today, and stage typed ops too. Staging is a parallel issue.
• The read model applies commits from the store, and applying one twice changes nothing. Applying a commit this process just made rewrites the same bytes and the same catalog rows, harmlessly.
• Moving repositories to materialiser-only writes, with the design's fitness rule against repository file writes, is a later step.

So this issue never calls into repositories, and the commit-log unit-of-work issue running in parallel never calls into the read model.

Goal

Build the v3 read model. It brings a repo's local files and a v3 catalog up to date with commits.db, starting from a stored cursor. It also keeps an index of live records and their BlobRefs, which chunk GC will use later. It is dormant: nothing in production uses it.

Design

1. Port and implementation:
   • a ReadModel port in src/domain/datastore_v3/, with an implementation in src/infrastructure/datastore_v3/;
   • catchUp(signal) reads changesSince(cursor) in pages, applies each commit in order, and advances a durably stored cursor;
   • catchUpPartition(namespace, partition, signal) catches up one partition before a read, if the shape fits.
2. Applying a commit:
   • put: write the record at its RecordKey under the namespace's cache root. Use the inline bytes, or stream from ChunkStore.read. Write atomically (temp file, then rename). Contain paths with the existing safe-path helpers.
   • tombstone: remove the record's file, ignoring "not found". Remove directories only when they're empty.
   • data-version: upsert the catalog row for that name and version from DataMetadata.
   • lease: nothing to materialise.
   • Idempotent and restartable. Re-applying a commit, or a crash at any point followed by catchUp, ends in the same files and catalog. Advance the cursor only after a commit's effects are durable.
3. The v3 catalog. Keep it separate from _catalog.db so no v2 code changes, for example as _catalog_v3.db beside commits.db. Reuse CatalogStore's schema and API if that works without changing CatalogStore's behaviour for v2. Record the choice and its reasons in datastores.md. Data queries switching to it is the opt-in's job.
4. The live-record index: for each namespace and partition, each present record's RecordKey and its BlobRef (or inline marker). It is updated as commits are applied. Chunk GC (a later issue) will compute the set of chunks in use from it, so it must be exact: no record present without its ref, and no ref kept after a tombstone.
5. Fitness rules. Keep it dormant by extending integration/datastore_v3_rules_test.ts, so no production code outside the v3 directories imports it. Add a rule that the read model never writes inside commits.db's tables, so the commit store stays the only writer of commits.

Tests

• Integration tests on a real temp directory, with the SQLite commit store and the filesystem storage/1:
  • append commits straight to the store, then catch up;
  • assert the files (bytes) and catalog rows a v2 save, delete, rename and gc would have produced for the same changes;
  • cover inline and multi-chunk content, and namespaced and unnamespaced datastores.
• Idempotence: applying every commit twice gives the same result as applying it once.
• Crash safety: an error injected after a file write but before the cursor advances, followed by catchUp, gives the same result as a clean run.
• A property test (fast-check): random command sequences from the existing reference model in v3_test_helpers.ts, appended and caught up, give files and an index that match the model's present records.
• The index: it matches the present records after every step, including tombstones and re-puts.

Out of scope

• Making repositories stop writing files, and the commit-log unit of work (a parallel issue).
• Chunk GC, which will use this issue's index.
• The commit feed and Phase 5 peers.
• The outbox.
• Wiring and the opt-in.
• Any change to _catalog.db or to v2 code.

Done when

• The read model, the v3 catalog and the live-record index are on main, dormant, with these tests.
• The catalog and "who writes local files" decisions are recorded in datastores.md.
• All existing suites pass unchanged.

Blocked by: nothing (#3236 and #3237 have merged). It runs in parallel with the commit-log unit-of-work issue.