Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#2683 s3/gcs datastore: pullChanged overwrites dirty, unpushed cache files

Opened by stack72 · 9/29/2026

Summary

pullChanged in @swamp/s3-datastore and @swamp/gcs-datastore overwrites local cache files that are dirty but not yet pushed. The pull walk compares each local file with the remote index (size, then mtime, then sha256) and downloads the remote copy when they differ. It never consults localDirty or dirtyPaths, so it can't tell "a peer pushed a newer version" from "this process changed the file and hasn't pushed it yet". It assumes the remote is right.

The next pushChanged then finds the file equal to the index and skips it, so the local change is lost.

Where

  • s3: datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts, pullChanged walk (the toPull loop, around L2536-2610)
  • gcs: datastore/gcs/extensions/datastores/_lib/gcs_cache_sync.ts, the same walk in pullChanged (around L2440-2490)

Reproduction (s3 unit level, mock S3 client from s3_cache_sync_test.ts)

  1. Write data/@m/model/next/latest = 15 with markDirty, then pushChanged.
  2. Write latest = 16 and 16/raw locally with markDirty. Do not push.
  3. Change the remote index the way another writer would (add an unrelated entry), so the pull skips its fast path.
  4. pullChanged({ subdirs: ["data"] }): local latest goes back to 15.
  5. pushChanged: 16/raw uploads and remote latest stays 15.

Proposed fix

While localDirty holds, the pull skips any index entry under a path in dirtyPaths. Core marks data-name directories, so match by prefix. The local change wins until it's pushed. When the dirty set can't be trusted (bulkInvalidated, dirtyPathsOverflowed, or a restart with the set lost), skip every local file that differs from the index rather than overwrite it.

Context

Found while triaging swamp-club lab #2488, where the serve runtime data poller's pull lands between a run's local write and its push. The root cause of #2488 is in core (serve lets a poller pull run during a run's unpushed-write window) and is being fixed there. This issue is the extension-side protection: any pull that runs while the cache holds unpushed work loses that work today.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/29/2026, 2:19:48 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

system commented 9/29/2026, 2:19:48 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.