Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#2249 s3-datastore: data 'latest' pointer objects are never indexed, so deleting a data item leaves an orphaned latest object in S3

Opened by hammz · 9/17/2026

Summary

With @swamp/s3-datastore 2026.09.10.2, deleting a data item removes its version objects (<dataName>/<N>/metadata.yaml, <dataName>/<N>/raw) from S3 but leaves <dataName>/latest behind.

The latest pointer is uploaded to S3, but it never appears in the extension's _index/ shard for the model. Deletion detection in pushChanged (absence on disk means delete) only deletes indexed objects, so the pointer is never removed. pullChanged still downloads it, so every poller pull puts <dataName>/latest back into the local cache with no version directories.

Reproduction (local MinIO; swamp built with the swamp-club#2240 fix, which makes serve push after data delete)

  1. S3 datastore on MinIO, managedConfig: true, swamp serve --auth-mode none --log-level debug.
  2. swamp model method run repro-shell execute --input run='echo x' --server $S writes result/1/* and result/latest.
  3. swamp data delete repro-shell result --yes --server $S returns versionsDeleted: 1.
  4. List S3 within 1s: result/1/* is gone, and result/latest is still there with its original LastModified. The _index/data--command--shell--<modelId>.json shard has no result/ entries, and no latest entry for any data name, even before the delete.
  5. Within one poll cycle, serve logs Config poller: 1 file(s) updated, Access data poller: 1 file(s) updated and Runtime data poller: 1 file(s) updated. The cache result/ directory comes back holding only latest.

Reproduced twice, on data names result and renamed.

Impact

  • data get stays "not found". Core ignores a latest marker whose version directory is missing, and the next write to that name overwrites the pointer.
  • Every deleted data item leaves an orphaned object in S3 and an orphaned file in every serve cache.
  • The index and bucket drift apart, which can break verifier and compaction tooling that assumes the index covers the whole data tree.

Expected

latest pointer files are indexed like the other files under data/ (or removed as part of the data-name subtree), so a push removes them together with the version directories.

  • swamp-club#2240 (core: serve data handlers now push)
  • swamp-club#2246 (pullChanged ignores subdirs)

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.09.10.2
  • swamp: 20260917.034341.0-sha.81c00b86
  • OS: darwin (aarch64)
  • Deno: 2.9.6
  • Shell: /bin/zsh
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/17/2026, 5:02:33 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.