Skip to main content
← Back to list
01Issue
BugShippedExtensions
Assigneesstack72

Relationships

#867 Pre-sync pull aborts the whole command when an index entry's S3 object is missing — no built-in way to recover

Opened by mgreten · 6/27/2026· Shipped 6/29/2026

Thanks for all the recent datastore work — #666 and #788 have made a real difference here, and I really appreciate it. I ran into something I'm not fully sure about and wanted to share it in case it's useful; my diagnosis may well be incomplete, so please take the framing as a hypothesis rather than a confident root cause.

What I'm seeing

On a high-churn S3/MinIO namespace (agentic-tooling), every write-path command (data gc, model method run, datastore sync, datastore compact) was failing during its pre-sync pull with:

Failed to pull 9956 file(s) from S3: …/report-swamp-method-summary-json/874/metadata.yaml, …

The 9,956 files it was trying to pull are referenced in the remote .datastore-index.json but were not present in S3 (and not in the local cache). As far as I can tell the pull seems to treat every indexed entry as required, so a single missing object aborted the whole command before it did any work — including gc, which is the one thing that might have reconciled the index. It also meant a stuck write held the per-namespace lock for minutes, which I think was blocking concurrent runs.

My (uncertain) guess at how it got into this state

I think this might be fallout from a data gc run a few days ago — it looks like the S3 objects were deleted but the corresponding index entries weren't fully removed, leaving dangling references. I'm genuinely not certain of the sequence, and it's entirely possible something on my side contributed. The blast radius was contained: all 9,956 dangling entries were under a single adw-session instance, almost all of them report-swamp-method-summary report artifacts (the ones I mentioned growing in #811/#812). I diffed the index entries against an actual S3 listing of that instance — 9,956 were in the index but absent from S3, and 0 were the reverse.

How I worked around it (in case it helps reproduce)

I backed up the remote .datastore-index.json, removed exactly those 9,956 dangling entries (the ones with no matching S3 object), re-uploaded, and pulled — after that gc/sync/model method run all worked again. So the underlying data seems fine; it was specifically the index→S3 mismatch that wedged everything.

What might help (very much deferring to your judgment)

Would it make sense for the pull to treat a missing-object-for-an-indexed-entry as a recoverable condition — e.g. skip with a warning, or self-heal by dropping the dangling entry — rather than hard-failing the whole command? That would let gc/sync recover a namespace in this state on their own. A datastore sync --repair / --prune-dangling style escape hatch would work just as well if an implicit skip feels too magical. I don't have a strong opinion on the shape — just flagging that today there doesn't seem to be a built-in way back once an index references a missing object.

No urgency at all, and I'm very happy to be wrong about the cause. Glad to share the full file list, the before/after index, or test a build whenever it's useful.

Environment

  • swamp 20260626.202321.0-sha.1bf46b4f
  • @swamp/s3-datastore (MinIO backend)
  • namespace agentic-tooling; ~300k remote objects, lazy-hydration (local cache was ~3 objects)

Possibly related: #788 (gc↔S3 deletion), #829 (intra-namespace lock), #811 / #812 (method-summary artifact growth).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 2 MORETRIAGE+ 8 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

6/29/2026, 11:01:06 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack726/29/2026, 9:14:52 PM

Sign in to post a ripple.