#2856 Fitness: ratchet markDirty call sites, repository constructions outside the factory, and unhooked datastore writers

Background (read this first)

swamp is about to rework its datastore layer. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch. It is not merged; §2 of it is a write-path inventory.

How the datastore works today.

• Repositories write files into the repo's .swamp/ directory, or into the datastore cache dir for a custom datastore. They then call a mark hook, markDirty(relPath?), which reaches DatastoreSyncService (src/domain/datastore/datastore_sync_service.ts, interface at :168, the eight-rule markDirty contract at :199-257).
• A caller then flushes, through one of four paths:
  • acquireModelLocks().flush (src/cli/repo_context.ts:1723);
  • the global coordinator flushDatastoreSync (src/cli/mod.ts:2251);
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:45/153);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:83-97) under the sync gate.
• The S3 and GCS sync implementations live in swamp-extensions, not here. They own the dirty set, the .datastore-sync-state.json sidecar, _index shards and _meta.json.
• A filesystem datastore has no sync service at all (repo_context.ts:1170-1178).
• Each repo has a SQLite catalog, .swamp/data/_catalog.db, that DataQueryService reads.
• Serve runs pollers that pull peers' changes: Config, AccessData, RuntimeData.

What changes. The refactor lands on main in phases:

• Phase 1: a UnitOfWork port with a legacy adapter, so repositories stage writes instead of calling the hook directly.
• Phase 2: libswamp use cases own the unit of work, and CLI commands and serve stop calling markDirty/pushChanged themselves.
• Later phases: a new commit-log engine behind an opt-in format.

Phases 1 and 2 must not change any behaviour.

Why this issue exists. Before any of that lands, this repo's own tests have to pin today's behaviour, so a dropped mark, a lost delete or a missed catalog refresh fails a test instead of reaching users. An audit in September 2026 found:

• no in-memory remote to test sync against;
• no test that every file a repository changes gets marked, and several real unmarked paths;
• no two-repo propagation test;
• no ratchet on the write seams Phase 1 will move;
• the sync-service conformance suite only checks method shapes.

This issue is one of a set. The tracking issue is listed under Related. End-to-end CLI tests for the same risks are in swamp-club/swamp-uat #481-#485, #487, #490 and #492.

Rules (from AGENTS.md)

• Unit tests are in-process only: no subprocesses, no process-global mutation; use withMockedEnv.
• Integration tests in integration/ wire real components on a temp filesystem and must not spawn the CLI.
• Registry-registered test types use a per-run name built with crypto.randomUUID(), or invalidateType in a finally.
• No fixed sleeps (use waitFor from @swamp-club/swamp-testing), no wall-clock assertions, and Deno.utime for mtimes.
• Use @std/path for paths and assertPathEquals for path comparisons, since tests run on Windows too.
• New files need the AGPLv3 header (deno run license-headers).
• Pin today's behaviour, including known gaps. When today's behaviour is a bug, assert it as it is, with a comment naming the bug. Prefer an explicit pinned list checked with assertPinnedSet, so a later fix forces the test to be updated on purpose. Do not fix production code in these issues unless the issue says so.

Goal

Add architecture fitness ratchets on the write seams Phases 1 and 2 will move:

• direct markDirty/notifyDirty calls;
• datastore-tier repositories constructed outside the factory.

Pinning them now means no new seam can appear while the refactor is in flight, and each refactor PR shows its progress by shrinking the lists. Also add a rule that catches repositories constructed without a mark hook when they write to the datastore.

Current state

• integration/datastore_sync_rules_test.ts already has related rules:
  • :58 bans bare this.notifyDirty() in the seven PER_PATH_WIRED_REPOS;
  • :204 bans bare .markDirty() in src/serve/**;
  • :295 is a pinned ratchet on commands that take locks and cancel;
  • :381 requires resolved base dirs for Output and Evaluated repos;
  • :429 pins PINNED_REPO_LOCAL_AUTO_DEFINITION_READERS.
• It also has a reusable, comment-stripping constructorArgs(code, className) parser (:327-364) and top-level owner attribution (TOP_LEVEL_DECLARATION, :188-191).
• Ratchet API (integration/arch_fitness_helpers.ts):
  • productionSourceFiles(dir) is an async generator that skips _test.ts(x);
  • assertPinnedSet(actual, pinned, label, policy) does set equality. It fails on new entries and on entries that are gone.
• There is no ratchet on mark call sites or on repository construction.
• Counts today (non-test src/):
  • 64 direct markDirty/notifyDirty invocations: grep -rnE "\b(markDirty|notifyDirty)\??\.?\(" src --include='*.ts' --include='*.tsx' | grep -v "_test.tsx\?:" | grep -vE ":\s*(//|\*)" | grep -vE "(async |private |public )(markDirty|notifyDirty)\(" | wc -l. Adding markDirtyHook/markDirtyBulk makes 68.
  • 96 property references across 40 files, including passing repoContext.markDirty as a value.
  • 119 constructions of datastore-tier repositories outside repository_factory.ts, over these classes: YamlDefinition, FileSystemUnifiedData, YamlWorkflow, YamlWorkflowRun, YamlOutput, YamlEvaluatedWorkflow, YamlEvaluatedDefinition, YamlVaultConfig, JsonlAudit, JsonlVaultAudit, CompositeUnifiedData, Lockfile.
• Known hook gaps:
  • src/libswamp/models/evaluate.ts:146 and src/libswamp/workflows/evaluate.ts:174 construct YamlEvaluatedDefinitionRepository / the workflow equivalent without a mark hook. That makes swamp model evaluate / workflow evaluate datastore writes that are never marked, and the :381 rule only checks the second argument.
  • libswamp/models/create.ts:97, models/edit.ts:163 and workflows/create.ts:76 construct unhooked repos, but the caller sends a bare mark.

Work

Add integration/datastore_write_seams_rules_test.ts with three rules:

1. PINNED_MARK_CALL_SITES: every direct markDirty(/notifyDirty(/markDirtyHook(/markDirtyBulk( invocation in production code, keyed "<repo-relative file>: <top-level owner>". Do not key on line numbers. Policy text: "New code must not call markDirty directly; stage writes through the unit of work (datastore refactor Phase 1). If this is intentional, add it here with a reason."
2. PINNED_REPO_CONSTRUCTIONS: every new <DatastoreTierRepository>( outside src/infrastructure/persistence/repository_factory.ts, keyed "<file>: <owner>: <Class>". Policy text: construct through createRepositoryContext.
3. PINNED_UNHOOKED_WRITERS: constructions of hook-taking repositories (Definition, Workflow, WorkflowRun, EvaluatedDefinition, EvaluatedWorkflow, UnifiedData, Output) that pass no hook argument. Use constructorArgs to find the argument. Pin today's list, including the two evaluate sites above, with a comment on each saying whether the caller marks by hand (create/edit) or nothing marks (evaluate, a real gap).

Each list is sorted, with one short comment per group. Put the grep commands above in the test file header so counts can be reproduced.

Done when

The three rules pass on main with today's pinned sets, and adding a new direct call or construction fails with the policy text.

Dependencies

Blocked by: nothing, so this can start now
Blocks: nothing
Full dependency graph and waves: swamp-club#2865.

Related

• Tracking issue: swamp-club#2865 (https://swamp-club.com/lab/2865).
• swamp-club#2855 covers the behaviour behind these seams.