#2971 Datastore rework Phase 1: optional ambient unit of work in repositories (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer. The design is the commit-log proposal in design/enablers/datastore-commit-log.md on the datastore-rework branch (not merged), and the project plan is tracked in swamp-club#2865.

How writes reach the datastore today.

• Every datastore-tier repository writes files into the datastore cache (or the repo's .swamp/), then calls a private notifyDirty(absPath?). Each repository gets a MarkDirtyHook (src/domain/datastore/datastore_sync_service.ts:526, (relPath?: string) => Promise<void>) through its constructor.

• The seven hooked repositories, each with its own notifyDirty:

  • unified_data_repository.ts:156;
  • yaml_definition_repository.ts:135;
  • yaml_workflow_repository.ts:95;
  • yaml_workflow_run_repository.ts:93;
  • yaml_output_repository.ts:97;
  • yaml_evaluated_definition_repository.ts:104;
  • yaml_evaluated_workflow_repository.ts:123.

  All are in src/infrastructure/persistence/.

• createRepositoryContext (src/infrastructure/persistence/repository_factory.ts) passes the hook in. The CLI builds it with buildMarkDirtyHook (src/cli/repo_context.ts:224) when a custom datastore has a sync service (repo_context.ts:1099, :1271). buildMarkDirtyHook turns the absolute path into a forward-slash path relative to the cache (falling back to <repo>/.swamp), and drops paths outside both. No hook is built for filesystem datastores.

• The hook calls DatastoreSyncService.markDirty. Its eight-rule contract is at datastore_sync_service.ts:199-257. The rules that matter here:

  • it fires before the write;
  • a path that's absent on disk at push time means delete;
  • a call with no path means bulk, and bulk overrides per-path marks from the same operation;
  • calls are not deduplicated by core.

• Something else then pushes the marked changes, through one of four flush paths:

  • acquireModelLocks().flush → flushSinglePhasePush / flushTwoPhasePush (repo_context.ts:1514, :1841, :1941);
  • the global coordinator flushDatastoreSync in src/infrastructure/persistence/datastore_sync_coordinator.ts;
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:99);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:82).

Where this is going. The rework replaces "write a file, mark it, push later" with "commit a record through a unit of work". It lands in phases:

• Phase 1: repositories stage their changes through a UnitOfWork port, and a legacy adapter turns staged changes into exactly today's markDirty calls. No behaviour change.
• Phase 2: libswamp use cases open and commit the unit of work, and CLI commands and serve stop marking and pushing themselves.
• Phase 3 onward: a v3 adapter commits to a real commit log instead.

The safety net is already on main. All the pre-refactor tests from swamp-club#2865's Phase 1 gate have shipped:

• packages/testing/in_memory_remote.ts: an in-memory remote that reproduces the S3/GCS extensions, quirks included;
• src/infrastructure/testing/test_datastore_type.ts: registerTestDatastoreType;
• integration/repository_dirty_coverage_test.ts: every file a repository changes must be marked, with today's gaps pinned in KNOWN_UNMARKED;
• integration/datastore_write_seams_rules_test.ts: pinned lists of mark call sites, repository constructions and unhooked writers;
• integration/datastore_peer_propagation_test.ts;
• integration/datastore_remote_failure_test.ts;
• integration/usecase_sync_characterization_*_test.ts;
• packages/testing/datastore_conformance.ts: assertSyncServiceRoundTripConformance;
• src/cli/repo_context_property_test.ts;
• the swamp-uat datastore suite, which runs against PR binaries.

Rules for this work:

• Behaviour must stay the same. Every test above stays green with no changes to its expectations. A pinned gap list may shrink only if a test proves the gap really closed; it may never grow.
• Follow AGENTS.md and the ddd skill. Use the ddd skill for the design. Domain ports live in src/domain/, adapters in src/infrastructure/. Use named exports, no any, AGPLv3 headers (deno run license-headers), and no fire-and-forget promises.
• Tests follow AGENTS.md: unit tests in-process, integration tests in integration/ without spawning the CLI, no fixed sleeps, no wall-clock assertions, crypto.randomUUID() ids, per-run registry type names with invalidateType in finally.
• Run the verification workflows (verify-build, verify-reviews, verify-skills) before the PR, as AGENTS.md describes.

Goal

Let datastore repositories stage their changes into an ambient unit of work when one is active, and fall back to their mark hook when none is. This lets the three Phase 1 repository moves, and Phase 2's use cases, adopt the unit of work one piece at a time without changing what users see.

No production code opens a unit-of-work scope in this issue. Production behaviour is therefore identical by construction, and tests prove that staging inside a scope produces exactly the same marks as today.

Design

Ambient scope: src/infrastructure/persistence/unit_of_work_scope.ts

• runInUnitOfWork<T>(uow: UnitOfWork, fn: () => Promise<T>): Promise<T> runs fn with uow active.
• currentUnitOfWork(): UnitOfWork | undefined returns the active one.

Implement with AsyncLocalStorage from node:async_hooks, which already has precedents: src/infrastructure/persistence/pulled_extensions_lock.ts:87 and src/domain/models/process_trace_env.ts:56. It follows the async call chain, so concurrent operations in one process (serve handlers, Promise.all) never see each other's unit of work.

A unit of work belongs to one repository context

Two repository contexts often live in one process: integration tests run two repos side by side, and namespace migration touches more than one tree. A repository must only stage into an ambient unit of work created for its own context. Otherwise it would steal another repo's marks, and those changes would never reach the right datastore.

• When the legacy adapter is created, bind it to the MarkDirtyHook instance it wraps: add a target (or similar) to the adapter, not to the domain port.
• A repository stages into the ambient unit of work only when its own hook is that exact instance. Otherwise it calls its hook directly, as today.
• A repository with no hook (a filesystem datastore) has nothing to bind to. It skips the ambient unit of work and does what it does today, which is nothing.

One shared helper replaces the seven private notifyDirty bodies

Add signalChange(hook: MarkDirtyHook | undefined, change: StagedChange): Promise<void> next to the scope. In order:

1. If the ambient unit of work is bound to hook, await uow.stage(change).
2. Otherwise, if hook exists, await hook(path or undefined).
3. Otherwise resolve.

Then change each notifyDirty in the seven repositories to delegate to it:

• unified_data_repository.ts:156;
• yaml_definition_repository.ts:135;
• yaml_workflow_repository.ts:95;
• yaml_workflow_run_repository.ts:93;
• yaml_output_repository.ts:97;
• yaml_evaluated_definition_repository.ts:104;
• yaml_evaluated_workflow_repository.ts:123.

Inside notifyDirty, map its absPath? argument to a StagedChange:

• undefined → { kind: "bulk", reason: <method name> };
• a path → { kind: "write", path }.

notifyDirty can't tell a write from a remove today, and that's fine. The legacy adapter forwards both kinds identically, and the absence-on-disk rule decides deletes at push time. Telling them apart is the job of the later repository-move issues, which change call sites, not this one.

Do not change any notifyDirty call site, its timing (before the write), its order, or the constructor hook parameters. This issue changes how a signal is routed, never when or whether it is sent.

Tests

1. The main equivalence test. Parameterise integration/repository_dirty_coverage_test.ts so every row runs twice:

   • (a) as today, with no scope;
   • (b) inside runInUnitOfWork(createLegacyUnitOfWork(hook), …) for the harness's context.

   Assert:

   • the recorded mark sequence is identical between (a) and (b) for every row, in order and arguments;
   • in (b), the unit of work's staged() matches those marks one-to-one;
   • KNOWN_UNMARKED is unchanged.

   If parameterising the existing file makes it hard to read, add a sibling file that reuses its rows. Do not copy the row table.

2. Isolation (integration/unit_of_work_scope_test.ts):

   • Two repository contexts A and B in one process, each on its own in-memory remote (registerTestDatastoreType). Inside a scope bound to A, a write through B's repository goes to B's hook only, and A's unit of work stages nothing from B.
   • Nested scopes: the innermost wins, and the outer is restored after.
   • Two scopes running concurrently with Promise.all never see each other's changes.
   • A promise started inside a scope and awaited after it exits stays with that scope. Document this as the expected AsyncLocalStorage behaviour.

3. End-to-end equivalence on the real flush path. In integration/datastore_peer_propagation_test.ts and integration/datastore_remote_failure_test.ts, add at least one scenario each that runs the write inside a scope (with acquireModelLocks().flush unchanged). Assert that the remote's ops() and the peer's view match the existing no-scope scenario exactly.

4. Unchanged suites: all of these pass with no expectation changes:

   • usecase_sync_characterization_*_test.ts;
   • repo_context_test.ts;
   • the swamp-uat datastore suite. It runs on the PR binary, so CI exercises it.

5. Fitness: update integration/datastore_write_seams_rules_test.ts.

   • The seven notifyDirty bodies no longer call the hook directly, so their entries in PINNED_MARK_CALL_SITES move to the single signalChange helper. The list should shrink or stay the same size.
   • Keep the rule that bans bare this.notifyDirty() in PER_PATH_WIRED_REPOS (integration/datastore_sync_rules_test.ts).
   • Add a rule that production code outside tests does not call runInUnitOfWork yet, as a pinned empty list. Phase 2 will add entries on purpose, and this keeps Phase 1 provably inert.

Dependencies

Blocked by: swamp-club#2970, "UnitOfWork port and legacy adapter" (it needs UnitOfWork, StagedChange and createLegacyUnitOfWork). Blocks: the three Phase 1 repository moves, which are not filed yet:

• data and output repositories;
• definition, workflow and evaluated repositories;
• run, vault config and lockfile repositories.

Each will replace absolute-path notifyDirty calls with typed write, remove and bulk changes.

Done when

• The ambient scope and signalChange exist, and the seven repositories route through it.
• Every dirty_coverage row produces identical marks with and without a scope.
• The isolation, nesting and concurrency tests pass.
• The new fitness rule pins zero production runInUnitOfWork callers.
• Every existing datastore test and the swamp-uat datastore suite pass with no expectation changes.
• Verification workflows pass on the final commit.

Out of scope

• Opening scopes from use cases, CLI commands or serve (Phase 2).
• Removing the hook constructor parameters.
• Distinguishing write from remove at call sites (the repository-move issues).
• Fixing any gap pinned in KNOWN_UNMARKED.

Related

• Tracking: swamp-club#2865 (Phase 1).
• Port and adapter: swamp-club#2970.