Relationships
#3305 S3/GCS datastore sync: shared sync service lets one sync clear another's dirty marks and pull over unpushed files
Opened by stack72 · 10/12/2026
Follow-up to swamp-club#3146 (fixed in swamp core) and swamp-club#2421. The S3 and GCS datastore extensions still have four behaviours that let one sync in a process undo another's work when a single sync service instance is shared, as swamp serve does across concurrent runs. Core now re-marks, reads back and retries data versions written under a model lock, but it cannot protect other files, and its retries only narrow these windows.
Observed with @swamp/gcs-datastore at swamp-extensions 2aa11ed05 (datastore/gcs/extensions/datastores/_lib/gcs_cache_sync.ts; s3_cache_sync.ts has the same pattern):
commitPushclears the whole dirty set and writeslocalDirty: false(around lines 3734 and 3869), including marks added afterpreparePushwalked the set. The next push of the run that made those marks takes thetryFastPushChangedno-op (around 1856) and uploads nothing. Expected: clear only the paths the finishing push snapshotted, for example with a dirty generation counter.preparePushiterates the livethis.dirtyPathsSet (around 3487) across awaits. A concurrentcommitPushthat clears the Set ends the iteration early, so a push can upload alatestmarker but skip the version directory it names. Expected: iterate a snapshot of the set.preparePush's migration branches writelocalDirty: falsewhile the dirty set is non-empty (around 3424, 3434, 3446), andmarkSynceddecideskeepLocalDirtybefore an await and clears afterwards (around 1753-1779). Expected: keeplocalDirtywhile any mark is pending, decided at the moment of clearing.- A model-scoped
pullChangedwalks every entry of the shared index (around 2496-2549), narrowing only bysubdirs, and re-downloads any same-size file whose hash differs without consulting the dirty set. A pull for one model therefore reverts another model's unpushedlatestmarker, and also run records, evaluated definitions, output records and appended version content.removeIfUnchanged(around 4090) checks the dirty set by exact file path, so a directory mark does not protect the files inside it. Expected: a model-scoped pull walks only that model's entries, and no pull overwrites or deletes a path with a pending mark (directory marks covering their descendants).
Reproduction: swamp serve with the GCS datastore against fsouza/fake-gcs-server, 6 concurrent workflow runs per round over two command/shell models; with the released binary about 1 in 7 rounds lost a version (details and scripts in swamp-club#3146).
Open
No activity in this phase yet.
system commented 10/12/2026, 5:36:48 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.