Skip to main content
← Back to list
01Issue
BugShippedExtensionsPublic
Assigneesstack72

Relationships

#1626 migrate-index still drops shallow-path entries on 2026.08.07.1 — #1584 marked shipped but no released version carries the fix (self-contained repro)

Opened by sntxrr · 8/12/2026· Shipped 8/12/2026

Re-filing swamp-club#1584, which is marked shipped but is still present in the only published artifact. Not a complaint about the triage — I suspect the fix is merged and the label ran ahead of the release. This issue exists to carry a self-contained repro so the fix can be confirmed against a version number.

The status does not match any released artifact

evidence value
#1584 filed 2026-08-10
@swamp/s3-datastore latest in registry 2026.08.07.1
that version published 2026-08-07, three days before the report
re-pull of that version today byte-identical source
comments on #1584 none naming a fix version

For contrast, #1554 was closed with a comment naming @swamp/s3-datastore@2026.07.31.1, which I could pull and verify. There is no such version here.

Repro — no S3, no bucket, no credentials, no network

partitionKeyFromPath is a public static on the exported S3CacheSyncService, so the shipped code can be called directly. Save as repro.ts:

const modPath = Deno.args[0];
const { S3CacheSyncService } = await import(`file://${await Deno.realPath(modPath)}`);

const cases: Array<[string, string]> = [
  ["data/@vendor/model/<uuid>/<dataName>/<n>/raw",     "data/@v/m/1111/current/3/raw"],
  ["outputs/@vendor/model/<method>/<uuid>-<ts>.yaml",  "outputs/@v/m/sync/1111-2026-01-01T00-00-00-000Z.yaml"],
  ["definitions-evaluated/@vendor/model/<uuid>.yaml",  "definitions-evaluated/@v/m/1111.yaml"],
  ["auto-definitions/@vendor/model/<uuid>.yaml",       "auto-definitions/@v/m/1111.yaml"],
  ["workflow-runs/<workflowId>/<file>.yaml",           "workflow-runs/wf-1/workflow-run-abc.yaml"],
  ["workflows-evaluated/<file>.yaml",                  "workflows-evaluated/workflow-wf-1.yaml"],
  ["<root file>",                                      ".catalog-export.json"],
];

let failures = 0;
for (const [shape, sample] of cases) {
  const key = S3CacheSyncService.partitionKeyFromPath(sample);
  if (key === undefined) failures++;
  console.log(`  ${key === undefined ? "DROPPED" : "ok     "}  ${shape}  ->  ${key ?? "undefined"}`);
}
console.log(`${failures} of ${cases.length} shapes dropped`);
Deno.exit(failures === 0 ? 0 : 1);

Run it against the installed extension (the --allow-env is only because importing the module pulls in the AWS SDK):

deno run --allow-read --allow-env repro.ts \
  <repo>/.swamp/pulled-extensions/@swamp/s3-datastore/datastores/_lib/s3_cache_sync.ts

Actual output on @swamp/s3-datastore@2026.08.07.1

  ok       data/@vendor/model/<uuid>/<dataName>/<n>/raw          ->  data--@v--m--1111
  ok       outputs/@vendor/model/<method>/<uuid>-<ts>.yaml       ->  outputs--@v--m--sync
  ok       definitions-evaluated/@vendor/model/<uuid>.yaml       ->  definitions-evaluated--@v--m
  ok       auto-definitions/@vendor/model/<uuid>.yaml            ->  auto-definitions
  ok       workflow-runs/<workflowId>/<file>.yaml                ->  workflow-runs--wf-1
  DROPPED  workflows-evaluated/<file>.yaml                       ->  undefined
  DROPPED  <root file>                                           ->  undefined
2 of 7 shapes dropped

Exit code 1. Expected: 0 of 7 dropped.

Any entry returning undefined is discarded by groupEntriesByPartition via if (!key) continue; with no log line, so it never reaches a shard.

Root cause (unchanged from #1584)

static partitionKeyFromPath(rel: string): string | undefined {
  const segments = rel.split("/");
  if (segments.length < 2) return undefined;        // (a) drops every root-level file
  const subdir = segments[0];
  switch (subdir) {
    ...
    case "workflow-runs":
    case "workflows-evaluated": {
      if (segments.length < 3) return undefined;    // (b) wrong for workflows-evaluated
      return `${subdir}--${segments[1]}`;
    }

The two share a case arm but are different depths by design: workflow-runs/<workflowId>/<file> is 3 segments, workflows-evaluated/<file>.yaml is 2. Guard (b) therefore rejects 100% of workflows-evaluated entries. Guard (a) rejects any root-level file, of which .catalog-export.json is the notable one.

Production evidence that it has not self-healed

Same datastore as #1584, now 478 commits later (_meta.json v2, commitSeq=478, 104 partitions, 23,019 sharded entries):

STILL MISSING  .catalog-export.json                              (object present in S3, 8.6 MB)
STILL MISSING  workflows-evaluated/workflow-063a0cb1-....yaml
STILL MISSING  workflows-evaluated/workflow-e14e03ef-....yaml

Nothing re-derives them, so the gap is permanent until the partitioner changes.

Severity: low, fails safe

Repeating this so it is not over-prioritised: the S3 objects still exist — this is index-only. toDelete is computed from index entries, so a shorter index yields fewer deletion candidates, never more; this cannot cause data loss. Practical cost is that affected files look absent and get re-pushed.

The reason to fix it anyway is the silent continue. That is the same shape as the silently-skipped push in swamp-club#1557, which went unnoticed here for seven days. And .catalog-export.json is the catalog itself.

Suggested fix

  1. Split the case arm: workflows-evaluated requires >= 2 and partitions on segments[0]; workflow-runs keeps >= 3.
  2. Give root-level files a partition (e.g. _root) instead of dropping at (a).
  3. Replace the silent continue with a counter and a log line — or fail the migration outright. A migration that drops entries without saying so is worse than one that refuses to run.
  4. Assert sum(shard entries) === entryCount before writing _meta.json. That single check would have caught this at authoring time and would prevent the whole class.

Point 4 is the one I would most like to see, independent of the specific paths.

Ask

Which released version carries the #1584 fix? If it is merged but unreleased, this issue can just track the release. The repro above exits non-zero on the current artifact and will exit zero once it lands, so it doubles as the acceptance check.

Environment

  • CLI 20260809.004828.0-sha.b61c9de2
  • @swamp/s3-datastore@2026.08.07.1 (latest available)
  • Deno 2.8.3, darwin/aarch64

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.08.07.1
  • swamp: 20260812.013400.0-sha.278864d1
  • OS: darwin (aarch64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 2 MORECODE_CONFORMANCE_REVIEW+ 2 MORESESSION_SUMMARIZED

Shipped

8/12/2026, 6:40:53 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/12/2026, 6:15:28 PM
Editable. Press Enter to edit.

stack72 commented 8/12/2026, 6:44:06 PM

The fix from PR #187 is now published. Pull @swamp/s3-datastore@2026.08.12.1 and @swamp/gcs-datastore@2026.08.12.1 — the reporter's repro script should exit 0 against these versions.

sntxrr commented 8/14/2026, 5:13:39 PM

Confirmed fixed in @swamp/s3-datastore@2026.08.12.1. Thank you — that was a fast turnaround, and all four suggestions landed.

The repro from the issue body now exits 0 against the shipped artifact:

  ok  data/@vendor/model/<uuid>/<dataName>/<n>/raw     ->  data--@v--m--1111
  ok  outputs/@vendor/model/<method>/<uuid>-<ts>.yaml  ->  outputs--@v--m--sync
  ok  definitions-evaluated/@vendor/model/<uuid>.yaml  ->  definitions-evaluated--@v--m
  ok  auto-definitions/@vendor/model/<uuid>.yaml       ->  auto-definitions
  ok  workflow-runs/<workflowId>/<file>.yaml           ->  workflow-runs--wf-1
  ok  workflows-evaluated/<file>.yaml                  ->  workflows-evaluated
  ok  <root file>                                      ->  _root
0 of 7 shapes dropped

(was 2 of 7 on 2026.08.07.1)

partitionKeyFromPath splits the case arm and gives root files _root; groupEntriesByPartition collects dropped entries and warns naming up to five. The abort guard is stronger than what I suggested — I asked for an assertion, and you made it refuse to write an incomplete _meta.json:

[s3-sync] Migration aborted: N index entries but only M could be partitioned...

That is the right call. It turns this whole class from a silent gap into a loud failure.

One thing worth documenting: the fix is not retroactive

This is the part that could confuse someone upgrading with an index already migrated by the buggy version, so it may be worth a release note.

Upgrading alone does not backfill entries that were dropped by an earlier migration, and a normal push will not do it either:

$ swamp datastore sync --push
Pushing all local data to remote...
Pushed 0 files

The three entries stayed missing. The reason is that a scoped walk only considers paths in dirtyPaths; entries absent from the index are never candidates, so nothing re-adds them. Re-running migrate-index also does not help on a v2 index, because pullIndex({forceRemote:true}) assembles from the shards — which are exactly the thing missing the entries.

What did fix it was forcing a full walk (localDirty: true, bulkInvalidated: true, dirtyPaths: [] in the sync sidecar) and pushing:

Pushed 366 files

partitions:    106 -> 118      (_root and workflows-evaluated created)
index entries: 26556 -> 26905
commitSeq:     710 -> 711
.catalog-export.json                          PRESENT
workflows-evaluated/workflow-063a0cb1-....yaml PRESENT
workflows-evaluated/workflow-e14e03ef-....yaml PRESENT
S3 objects lost: 0

doctor datastores passes on both writers afterwards, including the new catalog-coverage check.

If you would rather users not hand-edit the sidecar, a --rebuild/--force flag on migrate-index that sources from the monolithic index (still retained) rather than the shards would be a cleaner recovery path. Low priority — the gap is index-only and fails safe, since toDelete is index-derived so a short index can only reduce deletion candidates.

Unrelated gotcha hit while validating

swamp extension pull @swamp/s3-datastore upgraded the on-disk extension to 2026.08.12.1 but left extensions/models/upstream_extensions.json pinned at 2026.08.07.1. My containerised swamp serve installs from that lockfile, so it would have silently kept running the buggy version, and an extension install would have downgraded the host back.

swamp extension update moves both, and swamp extension outdated reads the lockfile — so those are the reliable pair. If pull leaving the lockfile stale is intended, it might deserve a note in its help text; if not, happy to file it separately.

Verified on

  • CLI 20260813.161427.0-sha.03547878
  • @swamp/s3-datastore@2026.08.12.1
  • Namespaced S3 datastore, 26,905 index entries, 118 partitions, two concurrent writers

Sign in to post a ripple.