Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#3305 S3/GCS datastore sync: shared sync service lets one sync clear another's dirty marks and pull over unpushed files

Opened by stack72 · 10/12/2026

Follow-up to swamp-club#3146 (fixed in swamp core) and swamp-club#2421. The S3 and GCS datastore extensions still have four behaviours that let one sync in a process undo another's work when a single sync service instance is shared, as swamp serve does across concurrent runs. Core now re-marks, reads back and retries data versions written under a model lock, but it cannot protect other files, and its retries only narrow these windows.

Observed with @swamp/gcs-datastore at swamp-extensions 2aa11ed05 (datastore/gcs/extensions/datastores/_lib/gcs_cache_sync.ts; s3_cache_sync.ts has the same pattern):

  1. commitPush clears the whole dirty set and writes localDirty: false (around lines 3734 and 3869), including marks added after preparePush walked the set. The next push of the run that made those marks takes the tryFastPushChanged no-op (around 1856) and uploads nothing. Expected: clear only the paths the finishing push snapshotted, for example with a dirty generation counter.
  2. preparePush iterates the live this.dirtyPaths Set (around 3487) across awaits. A concurrent commitPush that clears the Set ends the iteration early, so a push can upload a latest marker but skip the version directory it names. Expected: iterate a snapshot of the set.
  3. preparePush's migration branches write localDirty: false while the dirty set is non-empty (around 3424, 3434, 3446), and markSynced decides keepLocalDirty before an await and clears afterwards (around 1753-1779). Expected: keep localDirty while any mark is pending, decided at the moment of clearing.
  4. A model-scoped pullChanged walks every entry of the shared index (around 2496-2549), narrowing only by subdirs, and re-downloads any same-size file whose hash differs without consulting the dirty set. A pull for one model therefore reverts another model's unpushed latest marker, and also run records, evaluated definitions, output records and appended version content. removeIfUnchanged (around 4090) checks the dirty set by exact file path, so a directory mark does not protect the files inside it. Expected: a model-scoped pull walks only that model's entries, and no pull overwrites or deletes a path with a pending mark (directory marks covering their descendants).

Reproduction: swamp serve with the GCS datastore against fsouza/fake-gcs-server, 6 concurrent workflow runs per round over two command/shell models; with the released binary about 1 in 7 rounds lost a version (details and scripts in swamp-club#3146).

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/12/2026, 5:36:47 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

system commented 10/12/2026, 5:36:48 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.