#2859 Tests: remote failures during sync keep local writes dirty and the next flush uploads them (every flush path)

Background (read this first)

swamp is about to rework its datastore layer. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch. It is not merged; §2 of it is a write-path inventory.

How the datastore works today.

• Repositories write files into the repo's .swamp/ directory, or into the datastore cache dir for a custom datastore. They then call a mark hook, markDirty(relPath?), which reaches DatastoreSyncService (src/domain/datastore/datastore_sync_service.ts, interface at :168, the eight-rule markDirty contract at :199-257).
• A caller then flushes, through one of four paths:
  • acquireModelLocks().flush (src/cli/repo_context.ts:1723);
  • the global coordinator flushDatastoreSync (src/cli/mod.ts:2251);
  • pushManagedConfigChanges (src/cli/managed_config_sync.ts:45/153);
  • serve's pushChangedToRemote (src/serve/handlers/shared.ts:83-97) under the sync gate.
• The S3 and GCS sync implementations live in swamp-extensions, not here. They own the dirty set, the .datastore-sync-state.json sidecar, _index shards and _meta.json.
• A filesystem datastore has no sync service at all (repo_context.ts:1170-1178).
• Each repo has a SQLite catalog, .swamp/data/_catalog.db, that DataQueryService reads.
• Serve runs pollers that pull peers' changes: Config, AccessData, RuntimeData.

What changes. The refactor lands on main in phases:

• Phase 1: a UnitOfWork port with a legacy adapter, so repositories stage writes instead of calling the hook directly.
• Phase 2: libswamp use cases own the unit of work, and CLI commands and serve stop calling markDirty/pushChanged themselves.
• Later phases: a new commit-log engine behind an opt-in format.

Phases 1 and 2 must not change any behaviour.

Why this issue exists. Before any of that lands, this repo's own tests have to pin today's behaviour, so a dropped mark, a lost delete or a missed catalog refresh fails a test instead of reaching users. An audit in September 2026 found:

• no in-memory remote to test sync against;
• no test that every file a repository changes gets marked, and several real unmarked paths;
• no two-repo propagation test;
• no ratchet on the write seams Phase 1 will move;
• the sync-service conformance suite only checks method shapes.

This issue is one of a set. The tracking issue is listed under Related. End-to-end CLI tests for the same risks are in swamp-club/swamp-uat #481-#485, #487, #490 and #492.

Rules (from AGENTS.md)

• Unit tests are in-process only: no subprocesses, no process-global mutation; use withMockedEnv.
• Integration tests in integration/ wire real components on a temp filesystem and must not spawn the CLI.
• Registry-registered test types use a per-run name built with crypto.randomUUID(), or invalidateType in a finally.
• No fixed sleeps (use waitFor from @swamp-club/swamp-testing), no wall-clock assertions, and Deno.utime for mtimes.
• Use @std/path for paths and assertPathEquals for path comparisons, since tests run on Windows too.
• New files need the AGPLv3 header (deno run license-headers).
• Pin today's behaviour, including known gaps. When today's behaviour is a bug, assert it as it is, with a comment naming the bug. Prefer an explicit pinned list checked with assertPinnedSet, so a later fix forces the test to be updated on purpose. Do not fix production code in these issues unless the issue says so.

Goal

Pin what happens when the remote fails during sync, all the way through the real flush paths:

• the local write survives;
• the path stays dirty;
• the next successful flush uploads it.

Phase 1's legacy unit-of-work adapter has to keep exactly these semantics, including today's asymmetry between the flush paths.

Current state

• Coordinator: src/infrastructure/persistence/datastore_sync_coordinator_test.ts covers the global coordinator:
  • pull failure throws an enriched error (:155);
  • push failure is warned and swallowed (:192);
  • timeouts (:222, :309) and lock release (:286, :334).
• Per-model flush: src/cli/repo_context_test.ts covers the per-model flush, which throws where the global coordinator swallows (repo_context.ts:1723-1774, 1928-1976):
  • preparePush failure (:2550);
  • commitPush failure (:2588).
• Managed config: push errors warn (managed_config_sync_test.ts:130, :297).
• Serve pollers: survive a thrown pull (config_poller_test.ts:265, access_data_poller_test.ts:213, runtime_data_poller_test.ts:179).
• Gap: nothing tests write → push fails → local files still there and still dirty → next push succeeds → remote has the file. The dirty set lives inside the S3/GCS extensions, so it can only be tested here with the in-memory remote.
• Hygiene: datastore_sync_coordinator_test.ts:719 spawns a child Deno process and asserts elapsed time, which breaks the AGENTS.md unit-test rules. Move it to integration/ and replace the timing assertion with a work-done assertion, or document why it is an exception, so new tests don't copy it.

Needs

The in-memory remote with failNext / offline from the fake-remote issue.

Test

integration/datastore_remote_failure_test.ts. For each flush path:

• acquireModelLocks().flush single-phase;
• two-phase preparePush / commitPush;
• the global coordinator flushDatastoreSync;
• pushManagedConfigChanges.

Do the following for each:

1. Write data through a repository. Inject a push failure.
2. Assert the error surface. Per-model: throws. Coordinator: warns and does not throw. Managed config: warns.
3. Assert the local file is intact and the path is still dirty: the fake's dirty set still contains it.
4. Clear the failure and flush again. The remote now has the file, and a second repo pulling sees it.
5. Pull failure: inject one on lock acquire. Assert the command-level error, and that no local state was changed.
6. Offline for a sequence: writes on A while the fake is offline, then back online. Everything that returned success is on the remote after one flush.
7. Two-phase specifics: failure between prepare and commit leaves the paths dirty and the lock released, and a retry commits once. Check the remote contents, with no duplicate versions.

Done when

Every flush path has the scenarios above, the per-model versus coordinator asymmetry is pinned with a comment, and the :719 test is moved or documented.

Dependencies

Blocked by: swamp-club#2854
Blocks: nothing
Full dependency graph and waves: swamp-club#2865.

Related

• Tracking issue: swamp-club#2865 (https://swamp-club.com/lab/2865).
• End-to-end counterpart: swamp-uat#489 (deferred).