#3279 Datastore rework Phase 3: opt in to v3 for new filesystem datastores (single writer), and wire v3 into CLI and serve (no v2 behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3, every change is an explicit commit in a single-writer SQLite commits.db, content is stored as content-addressed chunks, and each checkout's local files are rebuilt from commits.

Everything v3 needs is on main (a7855165). It's dormant: fitness rules in integration/datastore_v3_rules_test.ts keep production code from importing any of it. It's documented in design/enablers/datastores.md under the "v3 …" sections. In order:

• #3233, the domain model (src/domain/datastore_v3/): partitions, intents, ops, commits.
• #3236, the commits.db commit store (src/infrastructure/datastore_v3/sqlite_commit_store.ts).
• #3237, the storage/1 contract and ChunkStore, with FileSystemStorage.
• #3239, the read model: state in .swamp/_read_model_v3/, the catalog in _catalog_v3.db.
• #3240, the commit-log unit and binding registry: runInCommitLogRoot, and requireUnit, which makes unscoped writes on a v3 context throw.
• #3249, chunk GC.
• #3250, #3256 and #3257, every datastore repository staging typed changes:
  • the shared route is stageTyped and typedStagingActive;
  • withUnitOfWork ends a v3 unit according to decision 5;
  • a run never names refused data, through the onRefused and reconcileRefusedRuns mechanism.
• #3258, history compaction: compactHistory, CompactionPolicy, resolveRunArtifact.
• The format guard (#3189, #3191). Every datastore open reads the format marker (<path>/datastore-format.json on filesystem datastores). It refuses any format not in SUPPORTED_DATASTORE_FORMATS, which today is [2] (src/domain/datastore/datastore_format.ts). Released binaries therefore already refuse a format-3 datastore.

The Phase 3 decision: v3 is for new, empty filesystem datastores with a single writer.

• S3 and GCS need Litestream to share commits.db (Phase 4, decided in design/enablers/datastore-litestream.md) and the extensions' storage/1, which is on hold.
• One writer per datastore. Two clones writing one commits.db on a shared mount isn't supported. The writer lock with its fence is Phase 4's.
• Repositories keep writing local files and also stage typed changes; the read model applies commits idempotently.

Goal

Make v3 usable, in one change: a user can create a new filesystem datastore in v3 format and every swamp command and serve handler works against it. Nothing changes for v2 datastores. The opt-in and the wiring ship together. A marker without wiring would make a datastore its own binary refuses.

Design

1. The opt-in.
   • A datastore config key selects v3 for a filesystem datastore, for example format: 3 on FilesystemDatastoreConfig (src/domain/datastore/datastore_config.ts), set through swamp datastore setup. Name it, and record the choice.
   • It's accepted only for a new, empty datastore path, and refused otherwise with a clear message. Migrating v2 data is Phase 6.
   • S3, GCS and extension datastores asking for v3 are refused with a message saying v3 there arrives with Phase 4.
   • Setting it up writes the marker {format: 3, minReaderFormat: 3, writtenBy} before any v3 file. A crash before the marker leaves an empty v2-looking directory, never v3 files without a marker.
2. The guard. Add 3 to SUPPORTED_DATASTORE_FORMATS only together with the wiring. Every guarded entry point (see integration/datastore_format_guard_rules_test.ts) must then open the right engine for the format it reads. Older binaries keep refusing v3, which the existing guard tests cover.
3. Single writer. In Phase 3, refuse a second writing process on a v3 datastore, for example with an exclusive lock file at the datastore path that's held while writing. Read-only commands may share. Name what the Phase 4 writer lock and fence will replace.
4. Wiring on a v3 datastore.
   • Repository contexts. createRepositoryContext (src/infrastructure/persistence/repository_factory.ts) builds repositories with a hook that is marked requireUnit.
   • The CLI. libSwampContextForRepo and runCommandInRootUnit (src/cli/) open commit-log roots (runInCommitLogRoot) instead of legacy roots, with commitStore, chunkStore, namespace, writer identity and onRefused (reconcileRefusedRuns) for workflow runs.
   • Serve. handlerLibSwampContext (src/serve/handlers/shared.ts) and serve's own roots do the same: detached runs, background work, start-up and shutdown. #3056's entry-point sweep lists every place serve opens a root.
   • No v2 sync on v3. Every v2 sync path (pull, push, pollers, the coordinator, the sync gate) must be skipped or refused for a v3 datastore, with a test for each.
   • Reads. Before a command or handler reads, catch up the read model, or catch up the partition for a single-aggregate read. Data queries (data query, data get, the catalog-backed lists) read _catalog_v3.db on v3. catalogDbPath and its callers choose by format.
   • Tier rules. Honour a repository's configured directories and exclude when deciding a file's staging tier, which #3256 left to the opt-in.
   • Chunk GC and compaction. Wire ChunkCollector into datastore gc on v3, and run compactHistory on the writer by CompactionPolicy, for example at the end of a write command or on a serve timer. Record when.
   • Dormancy rules. Unpin exactly the new production edges, each with its reason, and keep everything else pinned.
5. Follow-ups collected from earlier reviews. They belong here; see the plan's Follow-ups tab on #2865:
   • Run artifacts. Resolve run-record artifacts through resolveRunArtifact (#3258), and add sha256 to new DataArtifactRefs. That closes the same-top-version re-save case.
   • Unscoped writes. Hooked repository writes stay out of CLI lockfile transactions on v3 (#3192): a write there would be unscoped and refused.
   • Bulk changes. Keep every remaining v2 bulk change path unreachable on v3. Each must be a typed change, or refused.
   • Not this issue: restoring "data version numbers are never reused" waits until repositories stop writing files, and stays a plan item.
6. The four stream directories (audit, telemetry, logs, files) use the seam from swamp-club#3278, "a v3 home for audit, telemetry, logs and files". This issue must not merge before that one.

The gate

• v2 doesn't change:
  • every v2 suite passes with no expectation edits;
  • the whole-suite route recorder (#3240, #3250) shows identical routing;
  • the v2 format-guard tests pass unchanged.
• v3 works end to end on a filesystem datastore, in-process (integration/, no CLI subprocesses):
  • for every libswamp write use case and serve handler: run it on a v3 repository context, then catch up a read model in a second checkout and assert identical files and catalog;
  • refusals surface as errors the user can act on;
  • a second writer is refused;
  • an older binary's guard refuses the datastore (simulate it with SUPPORTED_DATASTORE_FORMATS = [2]).
• Every v2 sync path refuses or skips on v3, with a test each.
• The opt-in refuses non-empty paths, and S3, GCS and extension datastores.

Out of scope

• S3, GCS and extension v3 (Phase 4, and the extension work).
• Multi-writer and team mode (Phase 5).
• Migrating v2 data (Phase 6).
• Making repositories stop writing files.
• The swamp-uat fs-v3 backend, filed after this ships.

Done when

• A new filesystem datastore can be created in v3 and every command and handler works on it, proven by the gate.
• v2 is byte-for-byte unchanged.
• The decisions (key name, single-writer lock, compaction trigger, read catch-up points) are recorded in datastores.md.

Blocked by: nothing to start. It must not merge before swamp-club#3278, filed in parallel.