#3250 Datastore rework Phase 3: shared typed staging route, and data and output repositories stage typed changes on v3 (no v2 behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3, every change is an explicit commit in commits.db, made through a commit-log unit of work from typed changes: an intent that says what the operation is, plus its content. Phase 3 builds this dormant: nothing reaches it until a user opts a new, empty datastore into v3.

What's on main (8453c3cc), documented in design/enablers/datastores.md:

• Today's staging (v2). Repositories report each change through signalChange(markDirty, change) (src/infrastructure/persistence/unit_of_work_scope.ts). StagedChange is write | remove (a path) or bulk. The binding registry (#3240, unit_of_work_binding.ts) decides where it goes:
  • into the ambient unit bound to the repository's hook;
  • otherwise to the hook directly (route 2, kept on v2);
  • and on a hook marked requireUnit (a v3 context), a write outside a bound unit is refused with UnscopedWriteRefusedError.
• The commit-log unit (#3240, src/infrastructure/datastore_v3/commit_log_unit_of_work.ts, opened with runInCommitLogRoot in commit_log_root.ts):
  • takes stageOp(TypedChange), where a TypedChange is { partition, op } or { partition, put: PendingPut }, and PendingPut content is bytes or a stream factory, inlined up to 8 KiB and stored as chunks above that;
  • refuses path-only stage with PathOnlyChangeRefusedError;
  • staged() is always []; read stagedOps();
  • refuses more than MAX_COMMIT_OPS (10,000) staged changes for one partition until a checkpoint.
• The domain model (#3233, src/domain/datastore_v3/):
  • partitionOf(), createIntent, modelRef(ModelType, id), and the ops (putOp, tombstoneOp, dataVersionOp with DataMetadata);
  • data intents data.save, data.delete/data.gc (with a version, or none for keys that have no version, such as the latest marker or the name directory) and data.rename;
  • output intents output.save and output.delete.
  • The domain-model section maps today's methods to ops. For example, a save is a put of the raw content, a put of metadata.yaml, a data-version op and the latest marker. A data rename is two data-version ops in one commit: the new name's version and a rename-marker version on the old name.
• The read model (#3239): applies commits to a checkout's files and to _catalog_v3.db. Its v2-oracle test shows the files and catalog match what v2 would hold for the same changes.
• The Phase 3 decision: on v3, repositories keep writing local files exactly as today and also stage typed changes. The read model applies commits, and applying one twice changes nothing.

This issue is the first of three repository staging groups. The other two (definition, workflow and evaluated; run and vault config) wait for the shared typed route this one adds.

Goal

1. Add the shared typed staging route that every repository uses.
2. Make the data and output repositories (unified_data_repository.ts, yaml_output_repository.ts) stage typed changes on v3, for every write they make.

v2 behaviour must not change at all.

Design

1. The shared route, in unit_of_work_scope.ts or beside it. A repository passes today's path change and a lazy builder for its typed changes. For example:

   signalChange(markDirty, pathChange, () => typedChanges)

   • When the bound ambient unit is a commit-log unit, the builder runs and its changes go to stageOp.
   • Otherwise everything behaves exactly as today, and the builder never runs. That keeps v2 cost and behaviour identical.
   • Keep the route in one place, so the next two groups only add builders.

2. A small intent and partition helper for repositories. It builds modelRef and the intents from what each repository already has: model type, model id, data name, version, output id. Repositories never assemble partition keys by hand (the existing fitness rule forbids it).

3. The data repository. Cover every write path, for example:

   • save, including multi-chunk content streamed from the written file rather than buffered whole;
   • delete(version) and delete of a whole name, with each seen version as its own versioned tombstone plus the latest-marker rewrite;
   • rename: the new name's save plus the old name's rename-marker version, one commit;
   • collectGarbage and prune, with versioned data.gc tombstones. When one partition's changes would pass MAX_COMMIT_OPS, split them across checkpoints; the unit refuses more until then. Show it with a test above the limit.
   • rollbackVersions and advanceLatestMarkers, and the deferred-write reclaim paths.

   The exact files each one writes are today's: stage typed changes for exactly those keys. Use the domain-model section's mapping, and extend it where a path isn't covered, recording what you add.

4. The output repository: output.save and output.delete puts and tombstones for the files it writes and removes.

5. Bulk changes. Some v2 paths stage bulk, for example rename cleanups. On v3 each must become explicit per-key typed changes. There are no prefix tombstones, and a write with no typed form is refused. List every v2 bulk in these repositories, and show what each becomes.

The gate

• v2 is unchanged.

  • Every suite (usecase characterization, root-unit, dirty-coverage, peer-propagation, the unscoped-write guard and the routing test from #3240) passes with no expectation edits.
  • A whole-suite route recorder shows identical route-1 and route-2 writes, same caller, change kind and test, before and after. #3240 did the same; its PR describes the approach.
  • The typed builder never runs on v2: add a test that a builder which throws is harmless on v2.

• v3 is correct end to end. For every write path above:

  1. run the repository on v3 in a commit-log root over a temp directory;
  2. let the unit commit to a real commits.db and filesystem storage/1;
  3. catch up a read model into a second directory;
  4. assert that its files (bytes) and _catalog_v3.db rows equal what the repository wrote in the first directory.

  Reuse the read model's v2-oracle helpers. A missed key, a wrong intent or a missing version then fails a test.

Tests

• The gate above, as integration tests.
• Unit tests for the helper and the shared route: the builder runs only under a commit-log unit; v3 refuses a write outside a bound unit; on v3 the unit is read through stagedOps(), never staged().
• A test above MAX_COMMIT_OPS for collectGarbage or prune.
• The v3 dormancy rules still pass. If any new import crosses into the v3 directories, pin it deliberately, with its reason.

Out of scope

• The other two staging groups: definition, workflow and evaluated; run and vault config. Run staging also owns keeping a run record from naming data whose partition was refused.
• Making repositories stop writing files.
• Chunk GC (a parallel issue).
• The opt-in, and switching data queries to _catalog_v3.db.

Done when

• The shared route, the helper, and data and output staging are on main, with the gate's evidence in the PR.
• The bulk mapping and any mapping additions are recorded in datastores.md.
• All v2 suites pass unchanged.

Blocked by: nothing (#3239 and #3240 have merged). It runs in parallel with chunk GC. The definition/workflow/evaluated and run/vault config staging issues follow once this merges.