#2970 Datastore rework Phase 1: UnitOfWork port and legacy adapter (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer. The design is the commit-log proposal in design/enablers/datastore-commit-log.md on the datastore-rework branch (not merged), and the project plan is tracked in swamp-club#2865.

How writes reach the datastore today.

• Every datastore-tier repository writes files into the datastore cache (or the repo's .swamp/), then calls a private notifyDirty(absPath?). Each repository gets a MarkDirtyHook (src/domain/datastore/datastore_sync_service.ts:526, (relPath?: string) => Promise<void>) through its constructor.

• The seven hooked repositories, each with its own notifyDirty:

  • unified_data_repository.ts:156;
  • yaml_definition_repository.ts:135;
  • yaml_workflow_repository.ts:95;
  • yaml_workflow_run_repository.ts:93;
  • yaml_output_repository.ts:97;
  • yaml_evaluated_definition_repository.ts:104;
  • yaml_evaluated_workflow_repository.ts:123.

  All are in src/infrastructure/persistence/.

• createRepositoryContext (src/infrastructure/persistence/repository_factory.ts) passes the hook in. The CLI builds it with buildMarkDirtyHook (src/cli/repo_context.ts:224) when a custom datastore has a sync service (repo_context.ts:1099, :1271). buildMarkDirtyHook turns the absolute path into a forward-slash path relative to the cache (falling back to <repo>/.swamp), and drops paths outside both. No hook is built for filesystem datastores.

• The hook calls DatastoreSyncService.markDirty. Its eight-rule contract is at datastore_sync_service.ts:199-257. The rules that matter here:

  • it fires before the write;
  • a path that's absent on disk at push time means delete;
  • a call with no path means bulk, and bulk overrides per-path marks from the same operation;
  • calls are not deduplicated by core.

• Something else then pushes the marked changes, through one of four flush paths:

  • acquireModelLocks().flush → flushSinglePhasePush / flushTwoPhasePush (repo_context.ts:1514, :1841, :1941);
  • the global coordinator flushDatastoreSync in src/infrastructure/persistence/datastore_sync_coordinator.ts;
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:99);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:82).

Where this is going. The rework replaces "write a file, mark it, push later" with "commit a record through a unit of work". It lands in phases:

• Phase 1: repositories stage their changes through a UnitOfWork port, and a legacy adapter turns staged changes into exactly today's markDirty calls. No behaviour change.
• Phase 2: libswamp use cases open and commit the unit of work, and CLI commands and serve stop marking and pushing themselves.
• Phase 3 onward: a v3 adapter commits to a real commit log instead.

The safety net is already on main. All the pre-refactor tests from swamp-club#2865's Phase 1 gate have shipped:

• packages/testing/in_memory_remote.ts: an in-memory remote that reproduces the S3/GCS extensions, quirks included;
• src/infrastructure/testing/test_datastore_type.ts: registerTestDatastoreType;
• integration/repository_dirty_coverage_test.ts: every file a repository changes must be marked, with today's gaps pinned in KNOWN_UNMARKED;
• integration/datastore_write_seams_rules_test.ts: pinned lists of mark call sites, repository constructions and unhooked writers;
• integration/datastore_peer_propagation_test.ts;
• integration/datastore_remote_failure_test.ts;
• integration/usecase_sync_characterization_*_test.ts;
• packages/testing/datastore_conformance.ts: assertSyncServiceRoundTripConformance;
• src/cli/repo_context_property_test.ts;
• the swamp-uat datastore suite, which runs against PR binaries.

Rules for this work:

• Behaviour must stay the same. Every test above stays green with no changes to its expectations. A pinned gap list may shrink only if a test proves the gap really closed; it may never grow.
• Follow AGENTS.md and the ddd skill. Use the ddd skill for the design. Domain ports live in src/domain/, adapters in src/infrastructure/. Use named exports, no any, AGPLv3 headers (deno run license-headers), and no fire-and-forget promises.
• Tests follow AGENTS.md: unit tests in-process, integration tests in integration/ without spawning the CLI, no fixed sleeps, no wall-clock assertions, crypto.randomUUID() ids, per-run registry type names with invalidateType in finally.
• Run the verification workflows (verify-build, verify-reviews, verify-skills) before the PR, as AGENTS.md describes.

Goal

Introduce the UnitOfWork domain port and a legacy adapter that implements it on top of today's MarkDirtyHook. This is the seam every later datastore phase builds on:

• Phase 1 repositories stage changes into it;
• Phase 2 use cases commit it;
• Phase 3 adds a second adapter that commits to the new commit log.

This issue adds the port, the adapter and their tests. No production code calls them yet, so nothing a user can observe changes.

Design

The port: src/domain/datastore/unit_of_work.ts

A unit of work collects the changes one business operation makes to the datastore, and decides when they become durable and visible to others. Use DDD terms in names and docs. Suggested shape (adjust names if the ddd skill review suggests better ones, but keep the semantics):

/** One change a repository is about to make in the datastore. */
export type StagedChange =
  | { kind: "write"; path: string }   // absolute path about to be created or overwritten
  | { kind: "remove"; path: string }  // absolute path (file or directory) about to be removed
  | { kind: "bulk"; reason: string }; // a mutation that cannot be attributed to one path

export interface UnitOfWork {
  /** Records a change. Called BEFORE the write begins (markDirty rule 1). */
  stage(change: StagedChange): Promise<void>;
  /** Makes staged changes durable and visible. */
  commit(): Promise<void>;
  /** The changes staged so far, in order. */
  staged(): readonly StagedChange[];
}

Paths are absolute, as repositories hold them today. Converting them to cache-relative form stays where it is now (buildMarkDirtyHook), so the conversion rules can't drift. reason on a bulk change is a short fixed string naming the operation (e.g. "rename tombstone"), for diagnostics only.

The legacy adapter: src/infrastructure/persistence/legacy_unit_of_work.ts

createLegacyUnitOfWork(markDirty: MarkDirtyHook | undefined, options?: { flush?: () => Promise<void> }):

• stage forwards immediately and in order:

  • write and remove → markDirty(path);
  • bulk → markDirty(undefined).

  It must not batch, deduplicate, reorder or defer. The markDirty contract depends on pre-write timing and on the order of bulk and per-path marks (rules 1 and 8). The S3/GCS extensions drop path marks that arrive after a bulk mark, and the in-memory remote reproduces that.

• With no hook (filesystem datastores), stage records the change and resolves; it sends nothing, exactly as notifyDirty does today.

• A rejected hook rejects stage with the same error. Today notifyDirty awaits the hook and lets its error propagate, so this must not change.

• commit calls options.flush if one was given and otherwise resolves. In Phase 1 nothing in production calls commit. The existing flush paths keep pushing as they do today, and Phase 2 wires commit to them. Document this in the JSDoc.

• staged() returns what was staged, for tests and later phases. A unit of work lives for one operation, so the list is bounded by that operation. It is never process-lifetime state (serve keeps its sync service for the process lifetime; see DatastoreSyncService memory rule 2).

• Re-use is refused: staging after commit throws a clear programming error ("unit of work already committed"). Today nothing could depend on re-use, so this changes no behaviour.

Where it is not used yet

Do not change any repository, use case, CLI command or serve handler in this issue. swamp-club#2971 wires repositories to it.

Tests

1. Unit tests (src/infrastructure/persistence/legacy_unit_of_work_test.ts):

   • write, remove and bulk produce exactly hook(path), hook(path) and hook(undefined), in staging order;
   • no hook → no calls and staged() still records the changes;
   • a hook rejection rejects stage with the same error object;
   • commit calls flush once, or resolves without it;
   • staging after commit throws.

2. Property test (src/infrastructure/persistence/legacy_unit_of_work_property_test.ts, fast-check). For any random sequence of staged changes, the hook calls the adapter makes are identical, in sequence and arguments, to calling the hook directly with the same paths (undefined for bulk). This is the "the adapter is a pure pass-through" invariant Phase 1 relies on.

3. Integration test (integration/legacy_unit_of_work_test.ts):

   • two in-memory remotes, createInMemoryRemote() from @swamp-club/swamp-testing, wired through registerTestDatastoreType and buildMarkDirtyHook exactly as requireInitializedRepo wires them;
   • the same file writes and removes, once through direct hook calls and once through the adapter;
   • remote.ops() and remote.pendingPush(cacheDir) must be identical, including after a bulk mark followed by path marks;
   • one case with twoPhaseSync and one without.

4. A reusable contract suite: write the behavioural expectations every UnitOfWork must meet as one exported function, for example assertUnitOfWorkContract(factory) in src/infrastructure/testing/unit_of_work_contract.ts (test-only module, like test_datastore_type.ts):

   • stage order is preserved;
   • staged() reflects stage calls;
   • commit-once;
   • errors propagate.

   Run it against the legacy adapter. The Phase 3 v3 adapter will run the same suite.

5. Architecture fitness: the port module imports nothing from src/infrastructure/ (the existing ddd_layer_rules_test.ts should already enforce this; confirm it covers the new file). integration/datastore_write_seams_rules_test.ts is unchanged: the adapter's single markDirty call is new, so add it to PINNED_MARK_CALL_SITES with a comment that it is the legacy adapter.

Dependencies

Blocked by: the Phase 1 gate in swamp-club#2865, now open. Every unlock test has shipped. Blocks: swamp-club#2971, "Optional ambient unit of work in repositories", and through it the three Phase 1 repository moves (data and output; definition, workflow and evaluated; run, vault config and lockfile), which are not filed yet.

Done when

• The port and the legacy adapter exist with JSDoc explaining the Phase 1, 2 and 3 roles.
• All four kinds of test above pass. The contract suite is exported for reuse.
• The PR diff touches no repository, use case, CLI command or serve handler.
• Every existing datastore test (the list in Background) passes with no expectation changes.
• Verification workflows pass on the final commit.

Out of scope

• Changing repositories (sibling issue).
• Calling commit from production code (Phase 2).
• Changing buildMarkDirtyHook's path conversion.
• Any v3 or commit-log code.

Related

• Tracking: swamp-club#2865 (Phase 1).
• Next: swamp-club#2971.