#3257 Datastore rework Phase 3: workflow run and vault config repositories stage typed changes on v3 (no v2 behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3, every change is an explicit commit in commits.db, made through a commit-log unit of work from typed changes: an intent that says what the operation is, plus its content. Phase 3 builds this dormant: nothing reaches it until a user opts a new, empty datastore into v3.

What's on main (897b64a0), documented in design/enablers/datastores.md:

• The shared typed staging route (swamp-club#3250, src/infrastructure/persistence/unit_of_work_scope.ts):

  • signalChange(markDirty, change, { typed: true }). It keeps today's path change, which is skipped only when a typed stager is ambient.
  • stageTyped(markDirty, build). Called after the write, it runs build only under a commit-log unit bound to the hook, and stages the result as one batch, which lands in one commit. On v2 it returns after one lookup and never runs build. On a v3 hook with no bound unit, it throws UnscopedWriteRefusedError.
  • typedStagingActive(markDirty). It gates any read a typed change needs before the write destroys it, for example the digest of a file being replaced, so v2 reads nothing extra.
  • The model. unified_data_repository.ts and yaml_output_repository.ts already stage this way, through ModelChanges (src/domain/datastore_v3/repository_changes.ts). Follow that pattern.

• The domain model (swamp-club#3233, src/domain/datastore_v3/): partitionOf(), createIntent, intentPartition, the ops and MAX_COMMIT_OPS (10,000 changes per partition per batch). Intents include:

  • run.create, run.update and run.delete, plus run-snapshot.save and run-snapshot.delete, all on run--{workflowId}--{runId};
  • definition.create, definition.edit and definition.delete, plus definition.rename-to and definition.rename-from, and evaluated-workflow.save and evaluated-workflow.delete, on config--{model|workflow|vault}--{name};
  • evaluated-definition.save and evaluated-definition.delete, on the model partition;
  • auto-definition.save, on definition--{id}.

  A definition rename spans two partitions, committed in order: a definition.rename-to put with expect: "absent" in the new name's partition, then a definition.rename-from tombstone with expect: "present" in the old one.

• The commit-log unit (swamp-club#3240, #3249, #3250):

  • one commit per partition per batch, with run partitions last;
  • retries keep the txnId;
  • a post-commit chunk check re-stores missing chunks from the content source;
  • staged() is [] on v3, so read stagedOps().

• The read model (swamp-club#3239): applies commits to a second directory and to _catalog_v3.db. #3250's end-to-end test compares the files it rebuilds with what the repositories wrote. Reuse it: integration/datastore_v3_repository_staging_test.ts.

• The Phase 3 decision: on v3, repositories keep writing local files exactly as today and also stage typed changes.

This issue is one of two repository staging groups that build on #3250's route. The other is swamp-club#3256 (definition, workflow and evaluated).

Order of work (agreed while #3256 was in review):

1. Workflow runs first, now. They overlap with nothing in #3256.
2. Vault config last, once #3256 has merged. Rebase onto main, then implement vault config with #3256's definition helper (src/infrastructure/persistence/config_staging.ts, definitionSaveBatches), widening its kind from "model" | "workflow" to include "vault". Write no rename or edit logic of your own: #3256 already does the managed-config partition, the expect: { sha256 } on edits, the two-partition rename and the cleanups.

Put your tests in their own files, for example integration/datastore_v3_run_staging_test.ts and integration/datastore_v3_vault_staging_test.ts, not in #3250's or #3256's staging test files, so the branches don't conflict.

Goal

Make the workflow run and vault config repositories (yaml_workflow_run_repository.ts, yaml_vault_config_repository.ts) stage typed changes on v3 for every write they make. v2 doesn't change at all.

Design

1. Workflow runs.

   • The run record: run.create, then run.update, run.delete.
   • Run snapshots: run-snapshot.save and run-snapshot.delete.
   • Both live in run--{workflowId}--{runId}.
   • Run GC and supersede: per-key tombstones, split into batches when one partition would pass MAX_COMMIT_OPS.

2. Run outcomes (decision 5 in "v3 domain model"): a run's unit commits its partitions when the run ends completed, failed, suspended or cancelled, and discards on an error or early stop. Check the run paths (execution_service.ts, the resume and cancel paths, serve's detached runs) against that, and test each outcome on v3.

3. A run record must never name data whose partition was refused. This was a follow-up from #3240. The commit-log unit commits run partitions last, but it doesn't hold a run back when a data partition in the same flush was refused. Close it here:

   • Option one: when a data partition the run's changes depend on is refused, the run record's batch for that flush isn't committed, or is committed with that step recorded as failed.
   • Option two: the run record's typed change only names versions that landed.

   Pick one, record why in datastores.md, and test it: a refused data commit (for example an expect-failed or version-live refusal) followed by the run record.

4. Vault config (after #3256 merges).

   • Vault configs are datastore-tier only under managed config.
   • Stage them with #3256's definitionSaveBatches and kind: "vault" (config--vault--{name}). That gives definition.* intents, expect: { sha256 } on edits read through typedStagingActive, and a rename as two partitions, all from #3256's code.
   • Secrets are never committed; they go to the vault provider. Make sure no typed change carries secret material, and test that.
   • Main now has swamp-club#3259, which creates a default local_encryption vault at repo init and stops persisting an absolute base_dir. Check that the config written at init stages correctly on v3, and that no absolute path lands in a commit.

5. Today's bulk changes in these repositories become explicit per-key typed changes. List them, and record the mapping in datastores.md.

The gate

v2 doesn't change:

• every v2 suite (usecase characterization, root-unit, dirty-coverage, peer-propagation, the unscoped-write guard, the routing test) passes with no expectation edits;
• a whole-suite route recorder shows identical route-1 and route-2 writes before and after (same caller, change kind and test). #3240 and #3250 did this, and their PRs describe the approach;
• the builders never run on v2.

v3 is right end to end. For every write path:

1. run the repository on v3 in a commit-log root;
2. commit to a real commits.db and filesystem storage/1;
3. catch up a read model into a second directory;
4. assert that its files (bytes) and _catalog_v3.db rows equal what the repository wrote.

Cover namespaced and unnamespaced datastores. Add mutation or "red" checks for the rules you add, as #3250 did: drop a staged change and show a test fails.

Out of scope

• The definition, workflow and evaluated repositories (swamp-club#3256), other than widening its helper's kind for vault.
• Making repositories stop writing files.
• The opt-in.

Done when

• These two repositories stage typed changes on v3, with the gate's evidence in the PR.
• The run-outcome rules are tested on v3.
• The "never name refused data" rule is closed and recorded.
• All v2 suites pass unchanged.

Blocked by: nothing for the workflow-run part (#3250 has merged). The vault-config part waits for swamp-club#3256 to merge.