Skip to main content
← Back to list
01Issue
BugShippedExtensionsPublicTeam
Assigneeszack

Relationships

#4583 Scoped push queues each key twice: every data file is uploaded twice and every delete is sent twice

Opened by beddari · 11/10/2026· Shipped 11/10/2026

This report was written by an AI agent (Claude) on behalf of a user, from a reproduction on a throwaway S3 server. Names below are placeholders.

Summary

In a scoped push, collectPushWork adds the files and index entries under each dirty path to toPush / toDelete without removing duplicates. Core marks a data-name directory and paths beneath it (version directories, latest) in the same command, so the same key is queued twice.

The result, on every method run: each data file is uploaded twice, and each deleted key is deleted twice. Nothing fails; the cost is the second transfer of every data file and one extra request per delete.

Versions

  • @swamp/s3-datastore 2026.11.09.1 (latest). 2026.10.24.2 queued the same duplicates when it was run against an older gateway; the code path is unchanged between the two.
  • swamp 20261110.035534.0.
  • S3 server: versitygw v1.11.0, posix backend, unversioned bucket, plain HTTP on loopback, path-style. swamp datastore setup reported healthy.

Steps to reproduce

From an empty directory and an empty bucket <bucket> at <endpoint>, with the built-in command/shell type:

swamp repo init --no-vault --tool none
swamp extension pull @swamp/s3-datastore@2026.11.09.1
swamp datastore setup extension @swamp/s3-datastore \
  --config '{"bucket":"<bucket>","prefix":"repo","region":"us-east-1","endpoint":"<endpoint>","forcePathStyle":true}'
swamp behavior create command/shell sh
SWAMP_S3_SYNC_TRACE=1 swamp behavior run sh execute --input run='echo hi'

Expected

Each changed file is uploaded once and each removed key is deleted once.

Actual

The trace of that run (second and later runs; the first also uploads the evaluated definition, once):

[s3-sync] preparePush.walk ... toPush=25 toDelete=0
[s3-sync] preparePush.upload ... pushed=13

The server's access log shows 25 PUTs for 13 keys: raw, metadata.yaml and latest of each data name are PUT twice, the output record once. Index shards, _meta.json and the lock objects are written once. With --skip-reports it is toPush=13, pushed=7.

The same holds for a model of one's own: a method that writes one resource gave toPush=19, pushed=10 in 10 of 10 runs, with each of the 9 data files PUT twice.

Deletes are doubled the same way. A method that writes a new version of a data name and then calls context.deleteResource on it gave toDelete=6, deleted=6 for 3 keys in 20 of 20 runs, and the server logged each of <n>/metadata.yaml, <n>/raw and latest deleted twice, in the same concurrent batch.

Does anything fail

Not on a server that answers a delete of a missing key with success, as the S3 API documents: in the 30 runs above every command exited 0 with no error or warning, the lock was released, and the remote index matched the bucket exactly.

It did fail on versitygw v1.8.0 and v1.9.0, which answered the loser of two concurrent deletes with 404 NoSuchKey (versity/versitygw#2571, fixed in v1.11.0). There the extension counted the 404 as a failed delete, the push threw before commitPush, swamp logged Failed to release locks during cleanup, and the command still exited 0 (swamp-club#2299). That was seen in 20 of 54 runs there, and it is how the duplicates were found. No data was lost; the next push repaired the index.

Not measured: the time the second transfers cost with large objects or over a real network, and what a versioned bucket keeps of the duplicate requests.

Reading of the code

datastores/_lib/s3_cache_sync.ts, 2026.11.09.1:

  • collectPushWork, scoped-walk branch: iterates snap.paths and, for each dirty path on its own, pushes into toPush (a file, or every file under a directory) and into toDelete (index entries under an absent path, or entries under a directory with no local file). Nothing removes a key already added through another mark. While the method ran, dirtyPaths in .datastore-sync-state.json held data/<type>/<id>/main, data/<type>/<id>/main/<n> and data/<type>/<id>/main/latest together, so the marks overlap by design.
  • preparePush and pushChanged: the delete loop sends each batch of toDelete through Promise.allSettled, so the two deletes of one key are concurrent. Any rejection becomes formatBatchFailure("delete", ...) and is thrown.
  • S3Client.deleteObject in s3_client.ts does not treat NoSuchKey / 404 as success, and neither does the delete loop. Read paths in s3_cache_sync.ts do check err.name === "NotFound" || err.name === "NoSuchKey".
  • In core, flushTwoPhasePush rethrows, and the command's finally logs it as Failed to release locks during cleanup.

Possible fix

In the datastore extension's push path: remove duplicate keys from the upload and delete lists (or skip a dirty path that an ancestor mark already covers), and count a delete answered with 404 / NoSuchKey as done. The first removes the duplicate requests on every server; the second keeps a push from failing on a server that answers a repeated delete with 404. I have not checked whether @swamp/gcs-datastore builds its lists the same way.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.11.09.1
  • swamp: 20261110.035534.0-sha.8325fd88
  • OS: linux (x86_64)
  • Deno: 2.9.8
  • Shell: /bin/zsh
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 4:09:24 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/10/2026, 3:16:03 PM
Editable. Press Enter to edit.

system commented 11/10/2026, 1:39:52 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

zack commented 11/10/2026, 4:09:26 PM

Thanks @beddari for reporting this! We shipped: Scoped pushes on S3 and GCS list a key once per overlapping dirty mark, so files are uploaded twice and removed keys are deleted twice in one concurrent batch; on versitygw the losing delete gets NoSuchKey and the push fails. De-duplicate the scoped walk in collectPushWork for both extensions (without losing the present-file guard for a directory mark's delete pass), and make S3Client.deleteObject count a NoSuchKey answer as deleted, as the GCS client already does for 404.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.