Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#2865 Tracking: test baseline required before the datastore refactor (commit-log design)

Opened by stack72 · 9/30/2026

What this is

This issue tracks the test baseline that has to be in place before any code for the datastore refactor lands. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch; it is not merged. Every issue listed here is self-contained and can be picked up by an agent with no other context.

The refactor, in one paragraph.

  • Today repositories write files locally and call markDirty, and a sync service (in the S3/GCS extensions) pushes and pulls files, _index shards and _meta.json.
  • The refactor moves to a commit log: repositories commit through a unit of work, content becomes immutable chunks, and in team mode the CLI forwards commits to swamp serve.
  • It lands on main in phases:
    • Phase 1: a UnitOfWork port with a legacy adapter.
    • Phase 2: use cases own the unit of work.
    • Phase 3: a v3 engine, dormant behind opt-in.
    • Phase 4: solo mode with SQLite + Litestream.
    • Phase 5: serve leader, commit feed, model leases.
    • Phase 6: migrate/claim.
    • Phase 7: default flip.
    • Phase 8: MongoDB.
    • Phase 9: remove v2.
  • Phases 1-2 must change no behaviour. These tests are what prove that.

Gate: ready to start Phase 1

Every issue in waves 1-3 below is merged and green in CI. A test that exposes a bug today is committed pinned or ignored, with a link to a filed bug, not left out.

Dependency graph

An issue can start once everything it is blocked by has merged. Within a wave, issues are independent and can run in parallel.

swamp repo (swamp-club)                  swamp-uat
-----------------------                  ---------
#2854 in-memory remote fake              #481 filesystem backend + v2-only tag
  |-> #2855 every changed file marked      |-> #483 CI: backend list on PRs ------+
  |-> #2857 two-repo propagation           |                                      |
  |-> #2859 remote failures                |    #482 helpers                      |
  |-> #2860 use-case marks and pushes      |      |                               |
  |     '-> #2863 pollers (deferred)       +------+-> #484 fix tests that         |
  '-> #2861 conformance suite                     |        can't fail             |
                                                  |-> #485 deletes across clones  |
#2856 fitness ratchets       (independent)        |-> #487 file writers           |
#2858 fs catalog staleness   (independent)        |-> #490 managedConfig          |
#2862 property test          (independent)        '-> #492 every command syncs    |
                                                                                  |
#2864 CI: run swamp-uat suite on swamp PRs  <------- blocked by swamp-uat#483 ----+

Soft link (coordinate, not blocking): swamp-uat#485 on the filesystem backend is expected to hit the stale-catalog bug that swamp-club#2858 characterises. Land #2858 first, or at the same time, so #485 can pin those cases with an ignore pointing at it.

Waves

Wave Issues Blocked by
1: start now, in parallel swamp-club #2854, #2856, #2858, #2862; swamp-uat #481, #482 nothing
2 swamp-club #2855, #2857, #2859, #2860, #2861 swamp-club#2854
2 swamp-uat #483 swamp-uat#481
2 swamp-uat #484, #485, #487, #490, #492 swamp-uat#481 and #482
3 swamp-club #2864 swamp-uat#483

Critical paths:

  • swamp-club#2854 → the five wave-2 swamp issues.
  • swamp-uat #481 + #482 → the five wave-2 test issues.
  • swamp-uat#481 → #483 → swamp-club#2864.

Every issue: what it is, and what it is blocked by and blocks

In scope now: swamp repo (unit, integration, fitness, conformance)

Issue What Blocked by Blocks
swamp-club#2854 In-memory remote datastore fake + recording sync service (packages/testing) none #2855, #2857, #2859, #2860, #2861, #2863
swamp-club#2855 Every file a repository changes is covered by a mark; today's gaps pinned #2854 none
swamp-club#2856 Fitness ratchets: mark call sites (64), repository constructions (119), unhooked writers none none
swamp-club#2857 Two repos on one remote: writes, deletes, renames, gc propagate #2854 none
swamp-club#2858 Stale catalogs on a shared filesystem datastore (characterise the known bug) none none (soft link to swamp-uat#485)
swamp-club#2859 Remote failures keep writes dirty; next flush uploads (every flush path) #2854 none
swamp-club#2860 Marks and pushes for every libswamp write use case, CLI and serve #2854 #2863
swamp-club#2861 Behavioural sync-service conformance suite #2854 none
swamp-club#2862 Property test for buildMarkDirtyHook none none
swamp-club#2864 CI: run swamp-uat's datastore suite against the PR binary swamp-uat#483 none

In scope now: swamp-uat (end-to-end, compiled binary)

Issue What Blocked by Blocks
swamp-uat#481 Shared-filesystem backend in getTestBackends(); v2-only tag none #483, #484, #485, #487, #490, #492 (and deferred #488, #493, #495, #496)
swamp-uat#482 Helpers: serve cluster per backend, network cut, two-binary runner, fixture models, clone pair none #484, #485, #487, #490, #492 (and deferred #488, #489, #491, #493, #494)
swamp-uat#483 CI: run the backend list on PRs; enable never-run tests; matrix task + require-backends mode #481 swamp-club#2864
swamp-uat#484 Fix tests that pass without checking what they name #481, #482 none
swamp-uat#485 Deletes stay deleted on every clone #481, #482 none (soft link to swamp-club#2858)
swamp-uat#487 File-writer data, large files, versions, CEL .path across clones #481, #482 none
swamp-uat#490 managedConfig propagation between clones and serve instances #481, #482 none
swamp-uat#492 Every state-changing command leaves nothing unsynced; killed commands leave no partial data #481, #482 none

Deferred: needed before a later phase, not before Phase 1

Issue What Blocked by Needed before
swamp-uat#488 Concurrent runs from two clones on one remote #481, #482 Phase 5 (leases)
swamp-uat#489 CLI behaviour with the remote unreachable #482 Phase 5 (offline rules)
swamp-uat#491 Multi-instance serve on S3/GCS #482 Phase 5 (leader, feed)
swamp-club#2863 Serve pollers with a real catalog #2854, #2860 Phase 5
swamp-uat#493 Setup and migrations on repos that already hold data #481, #482 Phase 3 / 6
swamp-uat#494 Two swamp versions on one datastore #482 Phase 3 (format guard) / 6
swamp-uat#495 Datastore suite on Windows and macOS #481 Phase 4 (SQLite on clients)
swamp-uat#496 Baseline remote datastore and serve performance #481 Phase 4 benchmark

Behaviour gaps the audit found (to be pinned by the tests above, not fixed by them)

  • Unmarked writes and deletes in unified_data_repository.ts (delete(version) does not mark the latest rewrite or the name dir; advanceLatestMarkers / rollbackVersions) and in the YAML repos' rename and pathsToTry cleanups. See swamp-club#2855.
  • swamp model evaluate / workflow evaluate construct evaluated repositories without a mark hook (src/libswamp/models/evaluate.ts:146, src/libswamp/workflows/evaluate.ts:174). See swamp-club#2856.
  • CLI workflow approve / reject / cancel never push, while serve does. See swamp-club#2860.
  • Serve nodes or clones sharing a filesystem datastore never refresh their catalog. See swamp-club#2858.
  • Serve pollers treat a void pull result inconsistently (Config invalidates; Access and Runtime do not). See swamp-club#2863.

Each can be filed as its own bug when its test pins it. They are not fixed as part of this baseline.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/30/2026, 11:13:43 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

stack72 commented 10/1/2026, 5:42:42 PM

Follow-up for swamp-extensions (from swamp-club#2861, PR swamp-club/swamp#2774): @swamp-club/swamp-testing now exports assertSyncServiceRoundTripConformance, an experimental behavioural round-trip suite for DatastoreSyncService (two instances on one backend; round-trip, push and pull deletes, bulk marks, failed push retry, two-phase, no-op pull, forward-slash paths). The S3 and GCS datastore extensions should adopt it once the package publishes. Pull-side deletes run only with expectPullDeletes, since S3/GCS pulls never delete local files today. Caveat to check on adoption: pull-nothing-new sets local mtimes to 2001 and expects a pull with nothing new to leave the files untouched; a backend that decides changed by comparing mtimes would fail it. Related follow-ups filed while doing this: swamp-club#2906 (bring the in-memory remote's defaults up to s3/gcs-datastore 2026.10.01.1) and swamp-club#2907 (a push that fails after uploading drops its recorded deletes).

stack72 commented 10/2/2026, 9:50:28 PM

Plan change: the lockfile moves from Phase 1 to Phase 2.

Phase 1's last repository move was planned as "vault config and lockfile". It is now vault config only, filed as swamp-club#2995.

Why the lockfile can't move in Phase 1:

  • Its mark is a publish signal, not a write signal. ManagedLockfileTransaction.publish (src/libswamp/extensions/managed_lockfile_transaction.ts) marks the lockfile and then requires the push to upload something (mustUpload).
  • That mark has to stay. A failed publish stays pending, and a later extension change retries it, sometimes in a process that wrote nothing.
  • Moving it would add marks. If LockfileRepository also staged its writes, every lockfile change would send an extra mark. That breaks Phase 1's rule that marks stay identical, and would change the use-case characterisation tests' expectations.
  • It doesn't block the Phase 1 ratchet. LockfileRepository never calls markDirty itself.

Phase 2 item: when the managed lockfile transaction becomes a unit-of-work commit, redesign the publish signal (mustUpload and the pending retry) and stage lockfile writes through the unit of work.

Phase 1 order is now:

  1. Move C1, swamp-club#2992: the workflow run repository.
  2. Move C2, swamp-club#2995: vault config.
  3. The ratchet step, which removes changeFor and the hook fallback in signalChange.

Sign in to post a ripple.