Relationships
#2865 Tracking: test baseline required before the datastore refactor (commit-log design)
Opened by stack72 · 9/30/2026
What this is
This issue tracks the test baseline that has to be in place before any code for the datastore refactor lands. The design proposal is design/enablers/datastore-commit-log.md on the datastore-rework branch; it is not merged. Every issue listed here is self-contained and can be picked up by an agent with no other context.
The refactor, in one paragraph.
- Today repositories write files locally and call
markDirty, and a sync service (in the S3/GCS extensions) pushes and pulls files,_indexshards and_meta.json. - The refactor moves to a commit log: repositories commit through a unit of work, content becomes immutable chunks, and in team mode the CLI forwards commits to
swamp serve. - It lands on main in phases:
- Phase 1: a UnitOfWork port with a legacy adapter.
- Phase 2: use cases own the unit of work.
- Phase 3: a v3 engine, dormant behind opt-in.
- Phase 4: solo mode with SQLite + Litestream.
- Phase 5: serve leader, commit feed, model leases.
- Phase 6: migrate/claim.
- Phase 7: default flip.
- Phase 8: MongoDB.
- Phase 9: remove v2.
- Phases 1-2 must change no behaviour. These tests are what prove that.
Gate: ready to start Phase 1
Every issue in waves 1-3 below is merged and green in CI. A test that exposes a bug today is committed pinned or ignored, with a link to a filed bug, not left out.
Dependency graph
An issue can start once everything it is blocked by has merged. Within a wave, issues are independent and can run in parallel.
swamp repo (swamp-club) swamp-uat
----------------------- ---------
#2854 in-memory remote fake #481 filesystem backend + v2-only tag
|-> #2855 every changed file marked |-> #483 CI: backend list on PRs ------+
|-> #2857 two-repo propagation | |
|-> #2859 remote failures | #482 helpers |
|-> #2860 use-case marks and pushes | | |
| '-> #2863 pollers (deferred) +------+-> #484 fix tests that |
'-> #2861 conformance suite | can't fail |
|-> #485 deletes across clones |
#2856 fitness ratchets (independent) |-> #487 file writers |
#2858 fs catalog staleness (independent) |-> #490 managedConfig |
#2862 property test (independent) '-> #492 every command syncs |
|
#2864 CI: run swamp-uat suite on swamp PRs <------- blocked by swamp-uat#483 ----+Soft link (coordinate, not blocking): swamp-uat#485 on the filesystem backend is expected to hit the stale-catalog bug that swamp-club#2858 characterises. Land #2858 first, or at the same time, so #485 can pin those cases with an ignore pointing at it.
Waves
| Wave | Issues | Blocked by |
|---|---|---|
| 1: start now, in parallel | swamp-club #2854, #2856, #2858, #2862; swamp-uat #481, #482 | nothing |
| 2 | swamp-club #2855, #2857, #2859, #2860, #2861 | swamp-club#2854 |
| 2 | swamp-uat #483 | swamp-uat#481 |
| 2 | swamp-uat #484, #485, #487, #490, #492 | swamp-uat#481 and #482 |
| 3 | swamp-club #2864 | swamp-uat#483 |
Critical paths:
- swamp-club#2854 → the five wave-2 swamp issues.
- swamp-uat #481 + #482 → the five wave-2 test issues.
- swamp-uat#481 → #483 → swamp-club#2864.
Every issue: what it is, and what it is blocked by and blocks
In scope now: swamp repo (unit, integration, fitness, conformance)
| Issue | What | Blocked by | Blocks |
|---|---|---|---|
| swamp-club#2854 | In-memory remote datastore fake + recording sync service (packages/testing) |
none | #2855, #2857, #2859, #2860, #2861, #2863 |
| swamp-club#2855 | Every file a repository changes is covered by a mark; today's gaps pinned | #2854 | none |
| swamp-club#2856 | Fitness ratchets: mark call sites (64), repository constructions (119), unhooked writers | none | none |
| swamp-club#2857 | Two repos on one remote: writes, deletes, renames, gc propagate | #2854 | none |
| swamp-club#2858 | Stale catalogs on a shared filesystem datastore (characterise the known bug) | none | none (soft link to swamp-uat#485) |
| swamp-club#2859 | Remote failures keep writes dirty; next flush uploads (every flush path) | #2854 | none |
| swamp-club#2860 | Marks and pushes for every libswamp write use case, CLI and serve | #2854 | #2863 |
| swamp-club#2861 | Behavioural sync-service conformance suite | #2854 | none |
| swamp-club#2862 | Property test for buildMarkDirtyHook |
none | none |
| swamp-club#2864 | CI: run swamp-uat's datastore suite against the PR binary | swamp-uat#483 | none |
In scope now: swamp-uat (end-to-end, compiled binary)
| Issue | What | Blocked by | Blocks |
|---|---|---|---|
| swamp-uat#481 | Shared-filesystem backend in getTestBackends(); v2-only tag |
none | #483, #484, #485, #487, #490, #492 (and deferred #488, #493, #495, #496) |
| swamp-uat#482 | Helpers: serve cluster per backend, network cut, two-binary runner, fixture models, clone pair | none | #484, #485, #487, #490, #492 (and deferred #488, #489, #491, #493, #494) |
| swamp-uat#483 | CI: run the backend list on PRs; enable never-run tests; matrix task + require-backends mode | #481 | swamp-club#2864 |
| swamp-uat#484 | Fix tests that pass without checking what they name | #481, #482 | none |
| swamp-uat#485 | Deletes stay deleted on every clone | #481, #482 | none (soft link to swamp-club#2858) |
| swamp-uat#487 | File-writer data, large files, versions, CEL .path across clones |
#481, #482 | none |
| swamp-uat#490 | managedConfig propagation between clones and serve instances | #481, #482 | none |
| swamp-uat#492 | Every state-changing command leaves nothing unsynced; killed commands leave no partial data | #481, #482 | none |
Deferred: needed before a later phase, not before Phase 1
| Issue | What | Blocked by | Needed before |
|---|---|---|---|
| swamp-uat#488 | Concurrent runs from two clones on one remote | #481, #482 | Phase 5 (leases) |
| swamp-uat#489 | CLI behaviour with the remote unreachable | #482 | Phase 5 (offline rules) |
| swamp-uat#491 | Multi-instance serve on S3/GCS | #482 | Phase 5 (leader, feed) |
| swamp-club#2863 | Serve pollers with a real catalog | #2854, #2860 | Phase 5 |
| swamp-uat#493 | Setup and migrations on repos that already hold data | #481, #482 | Phase 3 / 6 |
| swamp-uat#494 | Two swamp versions on one datastore | #482 | Phase 3 (format guard) / 6 |
| swamp-uat#495 | Datastore suite on Windows and macOS | #481 | Phase 4 (SQLite on clients) |
| swamp-uat#496 | Baseline remote datastore and serve performance | #481 | Phase 4 benchmark |
Behaviour gaps the audit found (to be pinned by the tests above, not fixed by them)
- Unmarked writes and deletes in
unified_data_repository.ts(delete(version)does not mark thelatestrewrite or the name dir;advanceLatestMarkers/rollbackVersions) and in the YAML repos' rename andpathsToTrycleanups. See swamp-club#2855. swamp model evaluate/workflow evaluateconstruct evaluated repositories without a mark hook (src/libswamp/models/evaluate.ts:146,src/libswamp/workflows/evaluate.ts:174). See swamp-club#2856.- CLI
workflow approve/reject/cancelnever push, while serve does. See swamp-club#2860. - Serve nodes or clones sharing a filesystem datastore never refresh their catalog. See swamp-club#2858.
- Serve pollers treat a void pull result inconsistently (Config invalidates; Access and Runtime do not). See swamp-club#2863.
Each can be filed as its own bug when its test pins it. They are not fixed as part of this baseline.
Open
No activity in this phase yet.
stack72 commented 10/1/2026, 5:42:42 PM
Follow-up for swamp-extensions (from swamp-club#2861, PR swamp-club/swamp#2774): @swamp-club/swamp-testing now exports assertSyncServiceRoundTripConformance, an experimental behavioural round-trip suite for DatastoreSyncService (two instances on one backend; round-trip, push and pull deletes, bulk marks, failed push retry, two-phase, no-op pull, forward-slash paths). The S3 and GCS datastore extensions should adopt it once the package publishes. Pull-side deletes run only with expectPullDeletes, since S3/GCS pulls never delete local files today. Caveat to check on adoption: pull-nothing-new sets local mtimes to 2001 and expects a pull with nothing new to leave the files untouched; a backend that decides changed by comparing mtimes would fail it. Related follow-ups filed while doing this: swamp-club#2906 (bring the in-memory remote's defaults up to s3/gcs-datastore 2026.10.01.1) and swamp-club#2907 (a push that fails after uploading drops its recorded deletes).
stack72 commented 10/2/2026, 9:50:28 PM
Plan change: the lockfile moves from Phase 1 to Phase 2.
Phase 1's last repository move was planned as "vault config and lockfile". It is now vault config only, filed as swamp-club#2995.
Why the lockfile can't move in Phase 1:
- Its mark is a publish signal, not a write signal.
ManagedLockfileTransaction.publish(src/libswamp/extensions/managed_lockfile_transaction.ts) marks the lockfile and then requires the push to upload something (mustUpload). - That mark has to stay. A failed publish stays pending, and a later extension change retries it, sometimes in a process that wrote nothing.
- Moving it would add marks. If
LockfileRepositoryalso staged its writes, every lockfile change would send an extra mark. That breaks Phase 1's rule that marks stay identical, and would change the use-case characterisation tests' expectations. - It doesn't block the Phase 1 ratchet.
LockfileRepositorynever callsmarkDirtyitself.
Phase 2 item: when the managed lockfile transaction becomes a unit-of-work commit, redesign the publish signal (mustUpload and the pending retry) and stage lockfile writes through the unit of work.
Phase 1 order is now:
- Move C1, swamp-club#2992: the workflow run repository.
- Move C2, swamp-club#2995: vault config.
- The ratchet step, which removes
changeForand the hook fallback insignalChange.
Sign in to post a ripple.