#3025 Datastore rework Phase 2: use cases open and commit the unit of work (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865; design: design/enablers/datastore-commit-log.md on the datastore-rework branch). Repositories stop marking files dirty for a later push. Instead, each operation stages typed changes into a unit of work and commits it. Phase 1 is complete (swamp-club#2970, #2971, #2979, #2980, #2992, #2995, #2996). Phase 2 makes the use cases own the unit of work. Like Phase 1, it must change no behaviour.

What Phase 1 left on main (4f843533):

• The port (src/domain/datastore/unit_of_work.ts): UnitOfWork with stage(change), commit() and staged(). StagedChange is write, remove or bulk.
• The legacy adapter (src/infrastructure/persistence/legacy_unit_of_work.ts):
  • createLegacyUnitOfWork(markDirty, { flush }) forwards each staged change straight to the hook, as markDirty(path) or markDirty() for bulk.
  • commit waits for marks in flight, then awaits flush if one is given, and spends the unit.
  • Staging into a committed unit rejects with "unit of work already committed".
  • legacyUnitOfWorkTarget(uow) returns the hook a unit wraps.
• The ambient scope (src/infrastructure/persistence/unit_of_work_scope.ts):
  • runInUnitOfWork(uow, fn) and currentUnitOfWork(), built on AsyncLocalStorage.
  • signalChange(markDirty, change) stages into the ambient unit only when legacyUnitOfWorkTarget(uow) === markDirty. Otherwise it calls the hook directly (route 2), or does nothing with no hook (route 3).
  • No production code opens a scope. PINNED_UNIT_OF_WORK_SCOPES in integration/datastore_write_seams_rules_test.ts is empty, so every hooked write marks through route 2 today.
• Every hooked datastore-tier repository stages typed changes through signalChange(this.<hook>, …). Two fitness rules keep it that way.
• The hook: createRepositoryContext (src/infrastructure/persistence/repository_factory.ts) passes one markDirty instance to every hooked repository, and exposes it as repoContext.markDirty. buildMarkDirtyHook in src/cli/repo_context.ts builds it, only when a custom datastore has a sync service.
• The "End of Phase 1" section of design/enablers/datastores.md lists what still marks by hand and how pushes happen. There are four flush paths:
  • acquireModelLocks().flush;
  • flushDatastoreSync;
  • pushManagedConfigChanges and pushManagedConfigPaths;
  • serve's pushChangedToRemote.

How a use case runs today:

• libswamp use cases are async generators (ctx: LibSwampContext, deps, input) that yield stream events ending in completed or error. An example is dataDelete in src/libswamp/data/delete.ts.
• The CLI and serve build ctx with createLibSwampContext (src/libswamp/context.ts). It carries only signal and logger, plus withTimeout and withSignal for child contexts. They build deps with a createXxxDeps factory, often passing repositories from repoContext, and drive the stream with consumeStream or result (src/libswamp/stream.ts).
  • CLI pattern (src/cli/commands/data_delete.ts): requireInitializedRepoUnlocked, then acquireModelLocks, createLibSwampContext, createDataDeleteDeps(…, repoContext.unifiedDataRepo, …), consumeStream(dataDelete(ctx, deps, input), …), and finally lockResult.flush().
  • Serve pattern (src/serve/handlers/data_handlers.ts): createLibSwampContext() per request, then createDataDeleteDeps, dataDelete, and pushChangedToRemote(ctx).

Carried-forward requirements from the Phase 1 reviews:

• Hook identity. A scope only collects a repository's changes when the unit wraps that repository's exact hook instance. A unit built over a wrapped or rebuilt hook silently collects nothing, though marks still reach the sync service through route 2. Units must be built from repoContext.markDirty itself, and tests must prove staged() is non-empty after a write.
• Late writes. A write that lands after its unit committed makes stage reject. In production that would fail a user's command, so this issue must not let it happen silently (see Work item 4).

Goal

Every libswamp use case that writes through datastore-tier repositories runs its whole operation inside a unit of work, and commits it when it completes. The use case owns that boundary: it is the application service, so it decides where the transaction starts and ends.

In this issue commit pushes nothing. flush stays undefined, so marks reach the sync service exactly as today, and the existing flush paths keep pushing where they push now. Later Phase 2 issues move the push into commit and remove the hand marks.

Design

1. A unit-of-work factory on LibSwampContext.
   • Add a member that opens a fresh unit for one operation, for example openUnitOfWork(): UnitOfWork. The domain port type is UnitOfWork.
   • createLibSwampContext takes an optional factory. The default opens a legacy unit with no hook, which stages and records but sends nothing, so contexts built without one behave exactly as today.
   • withTimeout and withSignal children carry the same factory.
2. A libswamp helper that runs a use-case stream inside a unit, for example withUnitOfWork(ctx, () => impl(ctx, deps, input)) in src/libswamp/.
   • It opens one unit, then drives the inner generator. Every next() call runs inside runInUnitOfWork(uow, …), so all code the generator runs between yields sees the scope; AsyncLocalStorage doesn't flow into a generator any other way.
   • It re-yields each event.
   • It commits once, after the inner stream yields completed and finishes.
   • It does not commit when the stream yields error, throws, or is closed early by the consumer. The unit is abandoned: its marks were already sent, exactly as today.
   • Nested use cases open nested units, and the innermost wins while it runs.
   • This helper module is the only production code that references runInUnitOfWork. Add it to PINNED_UNIT_OF_WORK_SCOPES with that reason.
3. Wrap each write use case.
   • The exported use case keeps its signature (ctx, deps, input) and returns withUnitOfWork(ctx, …) around its existing body. Its callers don't change.
   • A write use case is any use case whose deps save, delete, rename or otherwise write through a datastore-tier repository: data, model, workflow and evaluated definitions, runs, outputs, vault config.
   • Use the five integration/usecase_sync_characterization_*_test.ts files as the checklist, because they exercise every write use case through the CLI and through serve. Add any write use case they don't reach to the pinned list in item 6.
   • Don't wrap read-only use cases.
4. Production stays forgiving about late writes. Add an option to the legacy adapter, for example { flush, afterCommit: "reject" | "forward" }.
   • "reject" is today's behaviour and the default.
   • "forward" marks a stage that arrives after commit straight to the hook, as route 2 would, and logs it once at debug with the change's path. That way a write escaping its use case never fails a command.
   • Production factories use "forward". Every test that drives use cases uses "reject", so a late write fails the test and gets fixed instead of hidden.
5. Composition binds units to the exact hook.
   • Wherever the CLI or serve already has a repoContext, pass createLibSwampContext a factory built over repoContext.markDirty itself: () => createLegacyUnitOfWork(repoContext.markDirty, { flush: undefined, afterCommit: "forward" }).
   • Never wrap or rebuild the hook.
   • Add one shared helper for this, for example in src/cli/repo_context.ts for the CLI and alongside serve's request context. Don't add 200 inline lambdas.
   • Where no repoContext exists (commands built on requireRepoMarker, or use cases that build their own unhooked repositories), keep the default factory. Those writes have no hook to bind to, so nothing changes.
6. Pin the wrapped use cases. Add PINNED_TRANSACTIONAL_USE_CASES to integration/datastore_write_seams_rules_test.ts, listing every libswamp use case that returns withUnitOfWork(…), keyed <file>: <export>. A use case gaining or losing its unit then shows up in review.

Tests

1. Unit tests for withUnitOfWork:
   • the scope is active inside the generator across several yields and awaits;
   • it commits exactly once after completed;
   • it doesn't commit after an error event, a throw, or a consumer that stops early (return());
   • nested use cases stage into the inner unit;
   • two concurrent use cases (Promise.all) never share a unit;
   • child contexts from withTimeout and withSignal open units from the same factory.
2. Legacy adapter afterCommit tests: "reject" is unchanged. "forward" marks the late change once through the hook, logs it, and never rejects.
3. Equivalence, the most important test: all five usecase_sync_characterization_*_test.ts files pass with no expectation changes. Every recorded mark, push and remote state is identical.
4. Hook identity, which proves the units really collect:
   • Extend the characterization harness so every CLI and serve step that writes also captures the unit its use case opened, through a test factory using afterCommit: "reject".
   • Assert that staged() equals the marks the step recorded, one-to-one and in order, and is non-empty whenever the step marked anything.
   • A unit bound to the wrong hook fails this.
5. No late writes in covered paths: with "reject" in the harness, none of the characterised steps stages after commit.
6. These pass unchanged:
   • integration/repository_dirty_coverage_test.ts;
   • integration/datastore_peer_propagation_test.ts and integration/datastore_remote_failure_test.ts;
   • the unit-of-work and scope tests;
   • src/cli/repo_context_test.ts;
   • every libswamp use-case unit test;
   • the swamp-uat datastore suite.
7. Fitness:
   • PINNED_UNIT_OF_WORK_SCOPES holds exactly the helper module;
   • PINNED_TRANSACTIONAL_USE_CASES is populated;
   • the existing ddd and libswamp layering rules pass. libswamp may import the domain port, and its helper may import the scope module, as libswamp deps factories already import infrastructure. If a rule objects, move the helper rather than weakening the rule.

Dependencies

Blocked by: nothing. Phase 1 is complete.

Blocks the rest of Phase 2. None of these are filed yet:

• CLI commands stop marking and pushing;
• serve handlers commit through the unit of work;
• one flush path;
• removing signalChange's hook fallback;
• redesigning the lockfile publish.

Done when

• LibSwampContext opens units, and its children carry the factory.
• withUnitOfWork exists, is unit-tested, and is the only production scope opener.
• Every write use case returns withUnitOfWork(…) and is pinned.
• Composition binds units to repoContext.markDirty itself through one helper per side (CLI, serve).
• The characterization suites pass unchanged, and the new hook-identity assertions prove each writing step's unit staged exactly its marks.
• Verification workflows pass on the final commit.

Out of scope

• Making commit push (a non-undefined flush).
• Removing any hand mark or flush path, or removing signalChange's fallback or the hook constructor parameters.
• The lockfile.
• Fixing any KNOWN_UNMARKED gap.

Related

• Tracking: swamp-club#2865 (Phase 2, first item).
• Built on Phase 1: swamp-club#2970, #2971, #2979, #2980, #2992, #2995, #2996.