#2855 Tests: every file a datastore repository changes on disk is covered by a markDirty (pin today's unmarked paths)

Background (read this first)

swamp is about to rework its datastore layer. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch. It is not merged; §2 of it is a write-path inventory.

How the datastore works today.

• Repositories write files into the repo's .swamp/ directory, or into the datastore cache dir for a custom datastore. They then call a mark hook, markDirty(relPath?), which reaches DatastoreSyncService (src/domain/datastore/datastore_sync_service.ts, interface at :168, the eight-rule markDirty contract at :199-257).
• A caller then flushes, through one of four paths:
  • acquireModelLocks().flush (src/cli/repo_context.ts:1723);
  • the global coordinator flushDatastoreSync (src/cli/mod.ts:2251);
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:45/153);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:83-97) under the sync gate.
• The S3 and GCS sync implementations live in swamp-extensions, not here. They own the dirty set, the .datastore-sync-state.json sidecar, _index shards and _meta.json.
• A filesystem datastore has no sync service at all (repo_context.ts:1170-1178).
• Each repo has a SQLite catalog, .swamp/data/_catalog.db, that DataQueryService reads.
• Serve runs pollers that pull peers' changes: Config, AccessData, RuntimeData.

What changes. The refactor lands on main in phases:

• Phase 1: a UnitOfWork port with a legacy adapter, so repositories stage writes instead of calling the hook directly.
• Phase 2: libswamp use cases own the unit of work, and CLI commands and serve stop calling markDirty/pushChanged themselves.
• Later phases: a new commit-log engine behind an opt-in format.

Phases 1 and 2 must not change any behaviour.

Why this issue exists. Before any of that lands, this repo's own tests have to pin today's behaviour, so a dropped mark, a lost delete or a missed catalog refresh fails a test instead of reaching users. An audit in September 2026 found:

• no in-memory remote to test sync against;
• no test that every file a repository changes gets marked, and several real unmarked paths;
• no two-repo propagation test;
• no ratchet on the write seams Phase 1 will move;
• the sync-service conformance suite only checks method shapes.

This issue is one of a set. The tracking issue is listed under Related. End-to-end CLI tests for the same risks are in swamp-club/swamp-uat #481-#485, #487, #490 and #492.

Rules (from AGENTS.md)

• Unit tests are in-process only: no subprocesses, no process-global mutation; use withMockedEnv.
• Integration tests in integration/ wire real components on a temp filesystem and must not spawn the CLI.
• Registry-registered test types use a per-run name built with crypto.randomUUID(), or invalidateType in a finally.
• No fixed sleeps (use waitFor from @swamp-club/swamp-testing), no wall-clock assertions, and Deno.utime for mtimes.
• Use @std/path for paths and assertPathEquals for path comparisons, since tests run on Windows too.
• New files need the AGPLv3 header (deno run license-headers).
• Pin today's behaviour, including known gaps. When today's behaviour is a bug, assert it as it is, with a comment naming the bug. Prefer an explicit pinned list checked with assertPinnedSet, so a later fix forces the test to be updated on purpose. Do not fix production code in these issues unless the issue says so.

Goal

Add a table-driven integration test proving that every file a repository creates, changes or deletes on disk is covered by a markDirty call. This is the main safety net for Phase 1: it moves those calls into a unit of work, and a dropped one silently leaves a change on one machine. Today's gaps get pinned as an explicit list, so the test is green now and any change, fix or regression, is deliberate.

Current state

• Per-repository unit tests check that a mark is forwarded. For example src/infrastructure/persistence/unified_data_repository_test.ts:752/894/943/1426, yaml_output_repository_test.ts:387, yaml_workflow_run_repository_test.ts:407 and yaml_definition_repository_test.ts:2001. None compares marks with the files that actually changed.
• integration/mark_dirty_hook_sync_test.ts:93 wires the real hook but asserts only a couple of cases.

Known unmarked paths today (found by reading the code; verify each while writing the test):

Test

integration/repository_dirty_coverage_test.ts

Setup (see integration/mark_dirty_hook_sync_test.ts:102-113):

• Build the repositories with createRepositoryContext({ datastoreResolver: new DefaultDatastorePathResolver(repoDir, { type: "@test/remote", config: {}, datastorePath, cachePath }), markDirty: buildMarkDirtyHook(recorder, cacheRoot, repoDir), … }), with the definitions and workflows dirs placed under the cache config/ dir. Otherwise the hook sends nothing and the rows prove nothing.
• Build YamlEvaluatedWorkflowRepository by hand; the factory does not create it.
• Use the recording sync service from swamp-club#2854.
• import "../src/domain/models/models.ts" for definitions.

The table. One row per write or delete method of the seven hooked repositories:

• Definition, Workflow, WorkflowRun, EvaluatedDefinition, EvaluatedWorkflow, UnifiedData, Output.
• Each row has a setup that creates prior state and an act that calls the method.
• Cover save, saveDeferred, append, allocateVersion + finalize*, delete (one version and all), removeLatestMarker, rename, collectGarbage, pruneExcessVersions, output delete/deleteOlderThan/deleteByMethodLifetime/deleteExpired/sweepOrphanLogs, run save/deleteAllByWorkflowId/deleteOlderThan, evaluated clearAll/clear/saveForRun/deleteForRun, and the rename paths of the YAML repos.
• For GC and age-based rows, backdate with Deno.utime.

Per row:

1. Walk the cache (@std/fs/walk, files only) before and after act, recording size and a hash. Ignore atomicWrite temp files.
2. Diff into added, removed and modified.
3. Assert that every changed path is covered by a mark: the exact relPath, or an ancestor directory relPath.
4. Assert no bare (bulk) mark was sent, unless the row is marked as expecting one.
5. Collect uncovered paths as "<Repository>.<method>: <relPath pattern>", with version numbers and ids normalised. Check them with assertPinnedSet (integration/arch_fitness_helpers.ts) against a KNOWN_UNMARKED list. A new gap fails, and a fixed gap fails until it is removed from the list.

Not in the table:

• YamlVaultConfigRepository and LockfileRepository: they take no hook, and callers mark by path (src/serve/handlers/vault_handlers.ts:838-846, admin_handlers.ts:1275-1286, managed_config_sync.ts:57). Cover them through the use-case characterisation issue.
• Audit repositories: always repo-local.
• The namespace manifest.

List these exclusions in a comment at the top of the test.

Done when

The test covers every write and delete method of the seven repositories and passes. KNOWN_UNMARKED holds exactly today's gaps, each with a comment pointing at the source line. The existing unified_data_repository_test.ts:851-856 pin is cross-referenced.

Dependencies

Blocked by: swamp-club#2854
Blocks: nothing
Full dependency graph and waves: swamp-club#2865.

Related

• Tracking issue: swamp-club#2865 (https://swamp-club.com/lab/2865).
• swamp-club#2856 pins the call sites and constructions this test exercises.
• End-to-end counterpart: swamp-uat#492.