Relationships
#2683 s3/gcs datastore: pullChanged overwrites dirty, unpushed cache files
Opened by stack72 · 9/29/2026
Summary
pullChanged in @swamp/s3-datastore and @swamp/gcs-datastore overwrites local cache files that are dirty but not yet pushed. The pull walk compares each local file with the remote index (size, then mtime, then sha256) and downloads the remote copy when they differ. It never consults localDirty or dirtyPaths, so it can't tell "a peer pushed a newer version" from "this process changed the file and hasn't pushed it yet". It assumes the remote is right.
The next pushChanged then finds the file equal to the index and skips it, so the local change is lost.
Where
- s3:
datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts,pullChangedwalk (thetoPullloop, around L2536-2610) - gcs:
datastore/gcs/extensions/datastores/_lib/gcs_cache_sync.ts, the same walk inpullChanged(around L2440-2490)
Reproduction (s3 unit level, mock S3 client from s3_cache_sync_test.ts)
- Write
data/@m/model/next/latest=15withmarkDirty, thenpushChanged. - Write
latest=16and16/rawlocally withmarkDirty. Do not push. - Change the remote index the way another writer would (add an unrelated entry), so the pull skips its fast path.
pullChanged({ subdirs: ["data"] }): locallatestgoes back to15.pushChanged:16/rawuploads and remotelateststays15.
Proposed fix
While localDirty holds, the pull skips any index entry under a path in dirtyPaths. Core marks data-name directories, so match by prefix. The local change wins until it's pushed. When the dirty set can't be trusted (bulkInvalidated, dirtyPathsOverflowed, or a restart with the set lost), skip every local file that differs from the index rather than overwrite it.
Context
Found while triaging swamp-club lab #2488, where the serve runtime data poller's pull lands between a run's local write and its push. The root cause of #2488 is in core (serve lets a poller pull run during a run's unpushed-write window) and is being fixed there. This issue is the extension-side protection: any pull that runs while the cache holds unpushed work loses that work today.
Open
No activity in this phase yet.
system commented 9/29/2026, 2:19:48 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.