Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#1874 migrate-index aborts on legacy bundles/ vault-bundles/ report-bundles/ entries — no in-product path forward

Opened by sbahra · 8/27/2026· Shipped 8/27/2026

swamp datastore migrate-index aborts on any repo whose datastore still holds objects under bundles/, vault-bundles/, or report-bundles/. These prefixes were written by the datastore itself in earlier swamp versions, are still present in the remote index, and have no case in partitionKeyFromPath — so they fall to default: return undefined and trip the migration's own completeness guard.

Distinct from #1584 / #1626 (shallow paths, fixed in this release). This is about top-level prefixes the datastore no longer manages but never removed.

Symptom

migrateIndex (s3_cache_sync.ts) partitions every index entry, then aborts if any entry has no shard:

if (partitionedCount !== entryCount) {
  throw new Error(`[s3-sync] Migration aborted: ${entryCount} index entries but only ${partitionedCount} could be partitioned. ...`)

On a repo initialized before these directories were dropped from the managed set, entryCount - partitionedCount is nonzero and migration can never run. The repo is stuck on the monolithic index permanently — in ours that is a 19.9 MB .datastore-index.json GET plus a 19.9 MB PUT on every mutating command.

Simulating the shipped partition logic over our live index:

total entries:     46652
would-be shards:     475
UNPARTITIONABLE:     217
     215  bundles/
       1  report-bundles/
       1  vault-bundles/

Repro — no S3, no bucket, no credentials, no network

Same technique as #1626; partitionKeyFromPath is a public static on the exported S3CacheSyncService. Save as repro.ts:

const modPath = Deno.args[0];
const { S3CacheSyncService } = await import(`file://${await Deno.realPath(modPath)}`);

const cases: Array<[string, string]> = [
  ["data/<vendor>/<model>/<uuid>/<dataName>/<n>/raw", "data/@v/m/1111/current/3/raw"],
  ["outputs/<vendor>/<model>/<method>/<uuid>-<ts>.yaml", "outputs/@v/m/sync/1111-2026-01-01T00-00-00-000Z.yaml"],
  ["workflow-runs/<workflowId>/<file>.yaml", "workflow-runs/wf-1/workflow-run-abc.yaml"],
  ["bundles/<hash>/<file>.js", "bundles/1bca5430/model.js"],
  ["vault-bundles/<hash>/<file>.js", "vault-bundles/7c811b12/vault.js"],
  ["report-bundles/<hash>/<file>.js", "report-bundles/037f1885/report.js"],
];

let failures = 0;
for (const [shape, sample] of cases) {
  const key = S3CacheSyncService.partitionKeyFromPath(sample);
  if (key === undefined) failures++;
  console.log(`  ${key === undefined ? "DROPPED" : "ok     "}  ${shape}  ->  ${key ?? "undefined"}`);
}
console.log(`${failures} of ${cases.length} shapes dropped`);

Run against the installed extension:

deno run --allow-read --allow-env repro.ts \
  <repo>/.swamp/pulled-extensions/@swamp/s3-datastore/datastores/_lib/s3_cache_sync.ts

Actual output on @swamp/s3-datastore@2026.08.26.1

  ok       data/<vendor>/<model>/<uuid>/<dataName>/<n>/raw  ->  data--@v--m--1111
  ok       outputs/<vendor>/<model>/<method>/<uuid>-<ts>.yaml  ->  outputs--@v--m--sync
  ok       workflow-runs/<workflowId>/<file>.yaml  ->  workflow-runs--wf-1
  DROPPED  bundles/<hash>/<file>.js  ->  undefined
  DROPPED  vault-bundles/<hash>/<file>.js  ->  undefined
  DROPPED  report-bundles/<hash>/<file>.js  ->  undefined
  3 of 6 shapes dropped

Expected

All six resolve to a shard key, and migrate-index completes.

Notes

  • swamp datastore status on our repo lists the managed dirs as auto-definitions, definitions-evaluated, workflows-evaluated, data, outputs, workflow-runs, audit, telemetry, logs, files. bundles is not among them, yet 215 bundle objects sit in the datastore, all written in a single burst a month ago and untouched since. So the guard is aborting on data the datastore wrote and then stopped managing.
  • Suggested fix: give unmanaged/unknown top-level prefixes a single-shard key (same treatment as audit/telemetry/logs), or have the guard skip prefixes outside the managed set rather than abort. Aborting leaves no in-product path forward — the only workaround is deleting objects out of the bucket by hand.
  • Workaround we are applying: move bundles/, vault-bundles/, report-bundles/ out of the datastore prefix, then migrate.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.08.26.1
  • swamp: 20260720.234931.0-sha.4770b0fd
  • OS: linux (x86_64)
  • Deno: 2.8.3
  • Shell: /bin/bash
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 2 MORETRIAGE+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/27/2026, 6:30:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/27/2026, 5:52:40 PM

Sign in to post a ripple.