#3034 Datastore rework Phase 2: serve handlers commit through the unit of work (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865; design: design/enablers/datastore-commit-log.md on the datastore-rework branch). Each operation will stage typed changes into a unit of work and commit it, instead of marking files dirty for a later push. Phase 1 is complete (swamp-club#2970, #2971, #2979, #2980, #2992, #2995, #2996), and so is the first Phase 2 step (swamp-club#3025). Phase 2 must change no behaviour.

On main (7f5a693a):

• The port (src/domain/datastore/unit_of_work.ts): UnitOfWork has stage, commit and staged. StagedChange is write, remove or bulk.
• The legacy adapter (src/infrastructure/persistence/legacy_unit_of_work.ts):
  • createLegacyUnitOfWork(markDirty, { flush, afterCommit }) forwards each change straight to the hook, as markDirty(path) or markDirty() for bulk.
  • commit awaits flush if one is given.
  • afterCommit: "forward" marks a late change instead of rejecting it.
• The scope (src/infrastructure/persistence/unit_of_work_scope.ts): runInUnitOfWork and currentUnitOfWork. signalChange stages into the ambient unit only when it wraps the repository's exact hook, and otherwise calls the hook directly ("route 2").
• Unit factories (src/infrastructure/persistence/repo_unit_of_work.ts):
  • openRepoUnitOfWork(markDirty) and repoUnitOfWorkFactory(repoContext) build production units with flush: undefined and afterCommit: "forward".
  • useUnitOfWorkFactoryForTesting is a test seam.
  • The CLI composes contexts with libSwampContextForRepo(repoContext, …) (src/cli/repo_context.ts), and serve with handlerLibSwampContext(ctx).
• Use cases (src/libswamp/unit_of_work.ts):
  • Every write use case (28, pinned in PINNED_TRANSACTIONAL_USE_CASES in integration/datastore_write_seams_rules_test.ts) runs inside withUnitOfWork(ctx, …).
  • The unit commits after completed, and is abandoned (nothing called) after error, a throw, an early return(), or another terminal. workflowRun can end on suspended or cancelled.
  • Today commit pushes nothing.
• How pushes happen today. Both sides push once per command or request, at the end, on every outcome:
  • CLI: acquireModelLocks(...) returns a lock whose flush() pushes and then releases. Commands call it in a finally, for example src/cli/commands/data_delete.ts (around line 338). Managed-config commands call pushManagedConfigChanges or pushManagedConfigPaths (src/cli/managed_config_sync.ts). The global coordinator's flushDatastoreSync pushes at process exit.
  • Serve: handlers call pushChangedToRemote(ctx) (src/serve/handlers/shared.ts) after their try/catch, inside the sync gate:
    • access handlers: 3 call sites;
    • admin handlers: 3;
    • data handlers: 5;
    • workflow handlers: 2;
    • src/serve/suspended_run_cancel.ts: 1.
• Hand marks still outside units are listed in PINNED_MARK_CALL_SITES, and in the "End of Phase 1" section of design/enablers/datastores.md.
• The equivalence suites:
  • the five integration/usecase_sync_characterization_*_test.ts files, through integration/usecase_sync_fixtures.ts. They record every mark, push and remote state, and assert that each use case's unit staged exactly its marks;
  • integration/repository_dirty_coverage_test.ts;
  • integration/datastore_peer_propagation_test.ts and integration/datastore_remote_failure_test.ts;
  • the swamp-uat datastore suite.

Rules:

• No behaviour change: the same marks (paths, bare or not, order) and the same pushes (count, timing relative to lock release, behaviour on failure). The suites above pass with no expectation changes unless an issue explicitly allows one.
• Follow AGENTS.md: named exports, no any, license headers, no fire-and-forget promises, no fixed sleeps, and the verification workflows before the PR.

Goal

Serve handlers commit through the unit of work. Each request that writes runs in a root unit of work (from swamp-club#3032) whose flush is pushChangedToRemote(ctx), run at the same point inside the sync gate, in place of the direct call. Serve's per-path hand marks become changes staged through the root. Marks and pushes stay identical.

Work

1. Handlers that call pushChangedToRemote(ctx) (src/serve/handlers/shared.ts):

   • access handlers: 3 call sites;
   • admin handlers: 3;
   • data handlers: 5;
   • workflow handlers: 2;
   • src/serve/suspended_run_cancel.ts: 1.

   Today each pushes after its try/catch, inside the handler's sync gate, so a failed request still pushes.

   • Wrap the handler's work in runInRootUnitOfWork(ctx.repoContext, { flush: () => pushChangedToRemote(ctx) }, async (root) => …), inside the existing gate.
   • Build the libswamp context with handlerLibSwampContext(ctx) inside the root, so use-case units become children.
   • Delete the direct pushChangedToRemote call.
   • Keep today's handling of a push failure: serve logs a warning and still answers the request. Use the helper's onFlushError, or the existing try/catch, so the response, its timing relative to the push, and the logs stay the same.
   • If a handler sends its response before pushing today, keep that order. Send inside fn and let the root flush after, and note it in the PR.

2. Workflow runs from serve go through executeWorkflowWithLocks (src/serve/deps.ts), used by the workflow handlers, src/serve/webhook.ts, the scheduler path in src/cli/commands/serve.ts, DispatchService and WorkerGateway.

   • Read how each pushes today. Where a push exists, make it the root's flush and wrap the run in a root.
   • Where none exists, don't add one. Leave it alone and say so in the PR.
   • A suspended or cancelled run must push exactly as it does today. Cover both in tests.

3. Per-path hand marks become root.stage({ kind: "write" | "remove", path }), with the same paths in the same order:

   • device auth: mintServerTokenImpl in src/serve/device_auth_handler.ts, 2 call sites;
   • grant tracking: publishGrantWrites in src/serve/grant_write_tracking.ts;
   • access reload: handleAccessReload in src/serve/handlers/access_handlers.ts.

   Where that code runs outside a request handler (start-up or background), open a root around it with the push it does today, or leave it pinned with the reason.

4. Leave these in place:

   • the extension lockfile (extensionLockfileTransaction in admin_handlers.ts and createDatastoreLockfileSync), deferred to its own Phase 2 issue;
   • the serve start-up definition migration in serve.ts;
   • the pollers.

Tests

1. The serve rows of all five usecase_sync_characterization_*_test.ts files run with the root-unit option from swamp-club#3032 and pass with no expectation changes: the same marks, the same push count and order relative to the response, and the same remote state. The harness checks that each root's staged() covers the row's marks.
2. Failure paths: a request whose use case fails still pushes once, and answers with the same error. Cover one data handler and one workflow handler, plus a suspended and a cancelled workflow run.
3. Concurrency: two concurrent requests through handleMessage get separate roots, and each pushes its own changes, under the gate as today.
4. Pins:
   • PINNED_MARK_CALL_SITES loses the serve hand marks this issue converts;
   • add a pinned list of remaining direct pushChangedToRemote callers, which should be empty if possible;
   • add a pinned list of serve entry points that open a root unit.
5. These pass unchanged: integration/datastore_remote_failure_test.ts, integration/datastore_peer_propagation_test.ts, the serve sync-gate tests (src/serve/sync_gate_test.ts), every serve handler test, and the swamp-uat serve and datastore suites.

Dependencies

Blocked by: swamp-club#3032 (commit pushes: root units). Runs in parallel with: swamp-club#3033 (CLI). Both edit usecase_sync_fixtures.ts, datastore_write_seams_rules_test.ts and the design doc's hand-mark list. Whichever lands second rebases and keeps both sides.

Done when

• Every serve handler and entry point listed above pushes only through its root unit's flush, inside the sync gate, at the same point as before.
• The converted hand marks are staged through the root.
• The characterization suites pass unchanged, with the root-unit checks switched on for serve rows.
• The pins and the design doc's hand-mark list are updated.
• Verification passes.

Out of scope

• CLI (swamp-club#3033).
• The lockfile.
• The pollers.
• One flush path.
• Replacing marks with more precise ones.
• Removing signalChange's fallback.

Related

• Tracking: swamp-club#2865 (Phase 2).
• Built on: swamp-club#3025 and Phase 1 (#2970, #2971, #2979, #2980, #2992, #2995, #2996).