Lab #2854: Test infra: shared in-memory remote datastore fake and recording sync service for pre-refactor datastore tests

Background (read this first)

swamp is about to rework its datastore layer. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch. It is not merged; §2 of it is a write-path inventory.

How the datastore works today.

Repositories write files into the repo's .swamp/ directory, or into the datastore cache dir for a custom datastore. They then call a mark hook, markDirty(relPath?), which reaches DatastoreSyncService (src/domain/datastore/datastore_sync_service.ts, interface at :168, the eight-rule markDirty contract at :199-257).
A caller then flushes, through one of four paths:
- acquireModelLocks().flush (src/cli/repo_context.ts:1723);
- the global coordinator flushDatastoreSync (src/cli/mod.ts:2251);
- pushManagedConfigChanges (src/cli/managed_config_sync.ts:45/153);
- serve's pushChangedToRemote (src/serve/handlers/shared.ts:83-97) under the sync gate.
The S3 and GCS sync implementations live in swamp-extensions, not here. They own the dirty set, the .datastore-sync-state.json sidecar, _index shards and _meta.json.
A filesystem datastore has no sync service at all (repo_context.ts:1170-1178).
Each repo has a SQLite catalog, .swamp/data/_catalog.db, that DataQueryService reads.
Serve runs pollers that pull peers' changes: Config, AccessData, RuntimeData.

What changes. The refactor lands on main in phases:

Phase 1: a UnitOfWork port with a legacy adapter, so repositories stage writes instead of calling the hook directly.
Phase 2: libswamp use cases own the unit of work, and CLI commands and serve stop calling markDirty/pushChanged themselves.
Later phases: a new commit-log engine behind an opt-in format.

Phases 1 and 2 must not change any behaviour.

Why this issue exists. Before any of that lands, this repo's own tests have to pin today's behaviour, so a dropped mark, a lost delete or a missed catalog refresh fails a test instead of reaching users. An audit in September 2026 found:

no in-memory remote to test sync against;
no test that every file a repository changes gets marked, and several real unmarked paths;
no two-repo propagation test;
no ratchet on the write seams Phase 1 will move;
the sync-service conformance suite only checks method shapes.

This issue is one of a set. The tracking issue is listed under Related. End-to-end CLI tests for the same risks are in swamp-club/swamp-uat #481-#485, #487, #490 and #492.

Rules (from AGENTS.md)

Unit tests are in-process only: no subprocesses, no process-global mutation; use withMockedEnv.
Integration tests in integration/ wire real components on a temp filesystem and must not spawn the CLI.
Registry-registered test types use a per-run name built with crypto.randomUUID(), or invalidateType in a finally.
No fixed sleeps (use waitFor from @swamp-club/swamp-testing), no wall-clock assertions, and Deno.utime for mtimes.
Use @std/path for paths and assertPathEquals for path comparisons, since tests run on Windows too.
New files need the AGPLv3 header (deno run license-headers).
Pin today's behaviour, including known gaps. When today's behaviour is a bug, assert it as it is, with a comment naming the bug. Prefer an explicit pinned list checked with assertPinnedSet, so a later fix forces the test to be updated on purpose. Do not fix production code in these issues unless the issue says so.

Goal

Build a shared, in-memory remote datastore fake with content, plus a shared recording sync service. The other pre-refactor tests build on it, and Phase 1's legacy UnitOfWork adapter will be tested against it.

Current state

There is no reusable fake. About 17 hand-rolled fake sync services are scattered through test files. Examples:
- recordingSyncService at src/cli/repo_context_test.ts:3406
- createRecordingSyncService at integration/mark_dirty_hook_sync_test.ts:62
- createRacingSyncService at integration/server_token_mint_sync_test.ts:201-231
The closest thing to a remote is createFakeDatastore() in src/serve/sync_gate_test.ts:400-437: a remote Set, a local Set and a dirty Set, with push treating a path absent locally as deleted remotely. It tracks paths only, not content.
packages/testing/datastore_test_context.ts:91 (createDatastoreTestContext) records markDirty relPaths (getSyncOperations()), but its push and pull do nothing and nothing uses it.

Work

Add createInMemoryRemote() in packages/testing/, next to createDatastoreTestContext, exported from the package so swamp-extensions can use it too. It returns a remote holding a Map<relPath, Uint8Array>, and a connect(cacheDir) that returns a DatastoreSyncService bound to that cache directory. Several instances share one remote, one per simulated machine.
Behaviour of each connected service:
- markDirty(relPath?) records the path; a call with no path sets a bulk flag. Follow the contract at datastore_sync_service.ts:199-257.
- pushChanged:
  - for each marked path (or every file under the cache when bulk), upload the file if it exists locally, otherwise delete it remotely;
  - clear the dirty set and bulk flag only on success;
  - return a count.
- pullChanged:
  - download remote files that are new or changed into the cache;
  - delete local files the remote dropped, since the last pull, that are not dirty locally;
  - return the count of changed files.
  - Make the delete behaviour an option (pullDeletes: true | false), so tests can reproduce either S3's or GCS's current behaviour. Document which one each extension does today, checked against swamp-extensions/datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts and gcs_cache_sync.ts.
- preparePush / commitPush: two-phase, following datastore_sync_service.ts:379/405. Prepare does not clear dirty state; commit applies the change and clears it.
- capabilities(): match the interface.
- Failure injection: failNext("push" | "pull" | "prepare" | "commit", error?) and offline(true | false).
- Operation log: ops() returns an ordered list of {instance, op, paths}, so tests can assert what was sent.
Add createRecordingSyncService() that records markDirty calls, including bare ones, and does nothing else. Replace two or three of the hand-rolled copies with it to prove the API fits. Do not migrate all 17 in this issue.
Registration helper: registerTestDatastoreType(remote) registers a per-run @test/remote-<uuid> datastore type in datastoreTypeRegistry whose provider returns remote.connect(cachePath). requireInitializedRepo then wires it end-to-end, following the pattern at src/cli/repo_context_test.ts:1004-1100. It returns a disposer that calls invalidateType.
Tests for the fake itself (*_test.ts next to it):
- round-trip between two instances;
- delete propagation;
- a failed push keeps the path dirty and the next push succeeds;
- bulk mark;
- two-phase;
- pullDeletes on and off.

Done when

The fake and helpers are exported from packages/testing, tested, and used by at least one existing test in place of a hand-rolled fake.

Dependencies

Blocked by: nothing, so this can start now
Blocks: swamp-club#2855, swamp-club#2857, swamp-club#2859, swamp-club#2860, swamp-club#2861, swamp-club#2863
Full dependency graph and waves: swamp-club#2865.

Tracking issue: swamp-club#2865 (https://swamp-club.com/lab/2865).

Session-start 'swamp model search --json' in the AGENTS.md template is unfulfillable - wire it to a hook or drop it

Flaky property test: analyzeExpression dot/bracket agreement fails when the generated model name is a CEL keyword

Accept usernames for user subjects in grant files

Docs: enable-managed-config should say to re-run config migrate when its push fails

datastore config migrate checks the sentinel in the stale local cache and re-runs a full migration over a shared S3 datastore

datastore config migrate writes its sentinel before pushing; after a failed push a re-run reports 'already completed' and never publishes

extension quality, fmt --check and push disagree: bare-specifier severity, cache hit skips gates, gate failure hides the rubric, fmt ignores project config

Docs: grant file reference shows a stale format and omits resources and subjects

Docs: extension push dry-run reference and how-to need the registry checks, API-call list and content hash from Lab #3016

Flaky property test: analyzeExpression hyphenated dot access fails when modelName generates a CEL keyword ("in")

extension quality and push crash with IsADirectory when additionalFiles lists a directory

extension push: a required global argument renders with a stray "" ("text": "string""")

Resolve non-admin actors in dashboard audit logs

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

Docs: extension publishing reference is stale on bare specifiers, quality output and fmt config (swamp-club#3022)

Docs: serve authorization page should cover expression-reference checks and the vault.get trust boundary

swamp-extensions: replace bare import specifiers with npm:/jsr: specifiers in first-party extensions

Docs: channel-aware extension push prompts and promote with a manifest (follow-up to #2939)

extension push: 'Requested visibility' line prints to stdout without the logger prefix, twice per run

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

extension push/quality: structured acceptances for agents, a clearer title than For next time, and an optional reason

Deliver workflow signals through a write-once outcome record instead of writing the run record

Shell steps: a vault.get on a continuation line of a multi-line double-quoted string inside a here-doc is wrapped in quotes

extension push: testing-completeness warns once per generated model (262 warnings in 25 extensions) with no way to mark a package generated

extension push: declare accepted lint warnings where the finding is (inline swamp-review-ignore comments, a sidecar review file beside the manifest), reported in quality, push and the registry

Docs: manual describes the old extension push path resolution (manifest argument, --extensions-dir scope, global skill fallback) after swamp-club#3018

extension push: --extensions-dir is ignored for bundled workflows

extension push: sub-directory extensions cannot be found from outside their directory (manifest path, directory argument, --extensions-dir for workflows and skills)

GCP bigquery datasets update cannot target the dataset: reads the id from a name field datasets do not have

extension push: the undecided private-entitlement verdict still says the registry may start a trial at publish

seed:collective cannot produce a superseded personal trial or a returned trial, so the UAT cannot cover them

Replayable state fragments: Git-reviewable desired state for resources changed by model methods

Extension bundler rewrites zod import text inside string/template literals, breaking code-generating extensions

s3-datastore: pull never removes files deleted on the remote, so a peer's next push re-uploads them and undoes `swamp data gc`

swamp-testing withMockedCommand: record env and cwd in CapturedCommandCall

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

extension push: registry metadata and content hash label files relative to the repo dir, so a push with --extensions-dir records ../ paths

Share dashboard authentication across tabs for deep links

Docs: vault.get shell references are placed per occurrence, and every single-quoted use is warned about

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Scheduled flaky-test detection is failing

Docs: declared acceptances for extension push warnings (quality.yaml sidecar, swamp-quality-ignore comments, For next time report)

Docs: describe the private-entitlement registry check and the refusal wording in the extension publishing reference

Fail fast when nested structural commands under parallel runs wait on each other's locks

Use whoami entitlement to explain private-extension push refusals

registry: store and show declared acceptances sent in contentMetadata.acceptances at confirm (swamp-club#3021)

@swamp/digitalocean/space-key: update and sync call /v2/spaces/keys/undefined

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

Deliver workflow signals through swamp serve

Docs: serve authorization of expression references, env and direct-type runs (swamp-club#2786)

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

extension push --dry-run: authentication verdict prints 'Signed in as .' when the credential is an API key with no username

Docs: dispatch-env-allow and how remote shell steps receive secrets (swamp-club#2760, swamp-club#2791)

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

Flaky test: WalSink replayed segments are delivered before events written after replay

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

serve --remote-only: step output from workers is never persisted to the run log (history logs always empty)

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

extension push --dry-run skips auth, collective and version-exists checks, and prints 'No API calls were made' while calling the registry

serve: a failed pull in acquireModelLocks leaves the per-model lock held until serve exits

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

serve HA: a server token minted on one replica is rejected by the other replicas until they restart (auto-definitions/ is only pulled at boot)

data_query_stale_limit_test fails on Linux: a limited page returns names in b, a order

extension push: version-drift check reports 'no previously published version found' when the registry call failed

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Search and list output follows directory iteration order in the definition, workflow, vault and output repositories

extension push: accept attestation-backed adversarial-review evidence (lookup by extension and content hash), generated provenance, and carry the evidence to the registry at confirm (design: Lab #3023)

verify-reviews: whole-extension rubric review for changed hand-written extensions, carried in the attestation and read by publish (design: Lab #3023)