Relationships
#4411 S3 and GCS pullFile truncates the local cache file in place, so an aborted or concurrent pull damages a live run record
Opened by skunk-ape · 11/4/2026
Problem
pullFile in the S3 and GCS datastore extensions opens the local cache file with truncate: true and streams the download into it. There is no temp file and no rename. While a download runs, the local file holds only part of its content, and an abort leaves that part behind for good.
The window has been wide since the streaming change of 2026-09-22 (swamp-extensions commit 3571a406d). Before it, the body was buffered first and written in one call.
A pull also replaces a local file whose size differs from the remote index entry. That includes the record of a workflow run still in progress, so a pull can put an older copy over a newer local record.
Evidence
- Stubbed S3 client, abort mid-download, on versions 2026.09.24.1 and 2026.11.03.1: a 305,599-byte local record was left as the first 262,144 bytes of the remote copy.
- Real run against MinIO on swamp 20261104.203255.0-sha.3ff9a703 with S3 extension 2026.11.03.1: 40 uncapped forEach iterations, while a second repo pushed to the same bucket and a second process ran
swamp datastore sync --pull. inotify recorded an in-place modify of the live run record, and the record went from 184,488 to 7,066 bytes. The next save restored it and the run succeeded. - GCS has the same
pullFilecode. This was confirmed by reading it, not by running it.
Expected
- A pull writes to a temp file beside the target and renames it over the target, so a reader or an abort never sees a partial file.
- A pull does not put an older remote copy over a local run record that has newer unpushed writes. The existing guard for marked paths applies only to subdir-scoped pulls.
Context
Found while diagnosing swamp-club#2517, where a truncated run record left a run permanently unreadable. The record damage there is not proven to come from this path. The core handling of a damaged record is fixed under #2517.
Triaged
Click a lifecycle step above to view its details.
system commented 11/4/2026, 11:30:02 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.