#2858 Tests: characterise stale catalogs when two repos or serve nodes share a filesystem datastore

Background (read this first)

swamp is about to rework its datastore layer. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch. It is not merged; §2 of it is a write-path inventory.

How the datastore works today.

• Repositories write files into the repo's .swamp/ directory, or into the datastore cache dir for a custom datastore. They then call a mark hook, markDirty(relPath?), which reaches DatastoreSyncService (src/domain/datastore/datastore_sync_service.ts, interface at :168, the eight-rule markDirty contract at :199-257).
• A caller then flushes, through one of four paths:
  • acquireModelLocks().flush (src/cli/repo_context.ts:1723);
  • the global coordinator flushDatastoreSync (src/cli/mod.ts:2251);
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:45/153);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:83-97) under the sync gate.
• The S3 and GCS sync implementations live in swamp-extensions, not here. They own the dirty set, the .datastore-sync-state.json sidecar, _index shards and _meta.json.
• A filesystem datastore has no sync service at all (repo_context.ts:1170-1178).
• Each repo has a SQLite catalog, .swamp/data/_catalog.db, that DataQueryService reads.
• Serve runs pollers that pull peers' changes: Config, AccessData, RuntimeData.

What changes. The refactor lands on main in phases:

• Phase 1: a UnitOfWork port with a legacy adapter, so repositories stage writes instead of calling the hook directly.
• Phase 2: libswamp use cases own the unit of work, and CLI commands and serve stop calling markDirty/pushChanged themselves.
• Later phases: a new commit-log engine behind an opt-in format.

Phases 1 and 2 must not change any behaviour.

Why this issue exists. Before any of that lands, this repo's own tests have to pin today's behaviour, so a dropped mark, a lost delete or a missed catalog refresh fails a test instead of reaching users. An audit in September 2026 found:

• no in-memory remote to test sync against;
• no test that every file a repository changes gets marked, and several real unmarked paths;
• no two-repo propagation test;
• no ratchet on the write seams Phase 1 will move;
• the sync-service conformance suite only checks method shapes.

This issue is one of a set. The tracking issue is listed under Related. End-to-end CLI tests for the same risks are in swamp-club/swamp-uat #481-#485, #487, #490 and #492.

Rules (from AGENTS.md)

• Unit tests are in-process only: no subprocesses, no process-global mutation; use withMockedEnv.
• Integration tests in integration/ wire real components on a temp filesystem and must not spawn the CLI.
• Registry-registered test types use a per-run name built with crypto.randomUUID(), or invalidateType in a finally.
• No fixed sleeps (use waitFor from @swamp-club/swamp-testing), no wall-clock assertions, and Deno.utime for mtimes.
• Use @std/path for paths and assertPathEquals for path comparisons, since tests run on Windows too.
• New files need the AGPLv3 header (deno run license-headers).
• Pin today's behaviour, including known gaps. When today's behaviour is a bug, assert it as it is, with a comment naming the bug. Prefer an explicit pinned list checked with assertPinnedSet, so a later fix forces the test to be updated on purpose. Do not fix production code in these issues unless the issue says so.

Goal

Add a characterisation test that pins a known bug. When two repos, or two swamp serve nodes, share one filesystem datastore, a repo whose catalog is already populated never sees the other's writes or deletes through catalog queries. Pinning it now means:

• the refactor doesn't make it worse without anyone noticing;
• the phase that fixes it flips the assertions on purpose;
• swamp-uat #485 and #481 (which add a shared-filesystem backend to the end-to-end suite) have a documented explanation when they hit it.

How it happens

1. A filesystem datastore gets no sync service (src/domain/datastore/datastore_config.ts:167-171, src/cli/repo_context.ts:1170-1178), and requireInitializedRepoUnlocked has no invalidation logic.
2. So serve creates no sync gate and no access or runtime pollers (src/cli/commands/serve.ts:2286, :3470-3489). The ConfigPoller only checks the lockfile hash.
3. lockResult.synced is always false, so the per-request catalogStore.invalidate() calls never fire.
4. The catalog is per repo at {repoDir}/.swamp/data/_catalog.db (src/infrastructure/persistence/repository_factory.ts:104-111), and its populated flag persists (catalog_store.ts:643-670).
5. Then:
   • DataQueryService.getLatestRecord returns null for names the other repo created (data_query_service.ts:256), or an old row for names it already knew (:229-241);
   • query never backfills (:392-393);
   • deleted rows are hidden only in query, through filterStaleRows (:659-688).

There is no issue, test or TODO for this today. The only write-up is §4 of the design doc.

Test

integration/filesystem_shared_datastore_catalog_test.ts

1. Two temp repos A and B with { type: "filesystem", path: <shared dir> } and the same namespace (or none), each built through requireInitializedRepoUnlocked as serve builds it. Use the writeOneItem pattern from integration/giga_swamp_namespace_layout_test.ts:81-118.
2. B writes one item and queries, so B's catalog is populated.
3. A writes a new name. Assert today's behaviour on B: getLatestRecord returns null, and query omits A's row.
4. A writes a new version of a name B knows. Assert B's getLatestRecord returns B's older row.
5. A deletes an item B has in its catalog. Assert query on B hides it (filterStaleRows), and record what getLatestRecord returns.
6. A control case. After catalogStore.invalidate() on B, B sees A's rows. This proves the only missing piece is invalidation.

Put a comment block at the top naming the source lines above, and say that the new datastore engine is expected to flip steps 3-5.

Done when

The test passes and documents the current behaviour. Also file a separate swamp-club Lab bug for the underlying behaviour if none exists, linked from the test comment.

Dependencies

Blocked by: nothing, so this can start now
Blocks: nothing
Full dependency graph and waves: swamp-club#2865.

Related

• Tracking issue: swamp-club#2865 (https://swamp-club.com/lab/2865).
• Soft link: swamp-uat#485 runs cross-clone deletes on the new filesystem backend (swamp-uat#481) and is expected to hit this bug. Land this first or at the same time.