Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublicTeam
Assigneesstack72

Relationships

Blocked

#1419 swamp doctor extensions: nondeterministic BundleBuildFailed for local extensions on an unchanged tree

Opened by magistr · 7/26/2026

Three consecutive swamp doctor extensions runs against an unchanged tree return exit 1, 0, 1.

Repro

In a repo with a local extension source that uses export const extension to add methods to a pulled model type:

for i in 1 2 3; do swamp doctor extensions >/dev/null 2>&1; echo "run $i: exit $?"; done
# run 1: exit 1
# run 2: exit 0
# run 3: exit 1

Observed

Failing runs report, for each export const extension source:

<path>.ts   BundleBuildFailed: Method "<name>" already exists on model type "<pulled type>"
<collective>  Indexed: 3 | BundleBuildFailed: 2
2 orphan bundle file(s)
3 passed, 1 failed — OVERALL: FAIL

The passing run logs Extension catalog updated: 2 "entries" repaired and reports all sources Indexed / OVERALL: PASS.

Why the message looks misleading

The pulled upstream does not declare the method names in question — grepping the pulled sources for each colliding name returns zero hits. So it does not appear to be a collision with upstream; the local extension seems to be colliding with itself, having already merged into the persisted catalog. In each case the colliding name is the first method in that extension's map, which reads like double-registration. Only the export const extension sources flap; export const model sources index on every run.

Runtime appears unaffected

swamp model type describe <type> --compact returns exit 0 with byte-identical output across consecutive runs, listing every method the extension contributes. So the resolved type surface is correct; only the indexing/bundling path races.

Impact

swamp doctor extensions is a natural deploy precondition — a bring-up step that runs it first and refuses to start on a load error will block at random. It also cannot be usefully baselined: an allowlist of "expect these N failures" cannot distinguish the expected N from a different N.

Workaround

Assert the property doctor is a proxy for — that the contributed methods resolve on their types via swamp model type describe — which is stable across runs.

Environment

swamp 20260725.011748.0-sha.bb2cb3ef, macOS. ~100 sources across 9 extensions; the affected local collective has 5 sources (3 export const model, 2 export const extension).

02Bog Flow
✓OPEN✓TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 2 MOREREVIEW

Closed

7/29/2026, 10:33:43 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack727/26/2026, 9:50:29 PM
stack72 marked as blocked7/26/2026, 11:13:51 PM
Editable. Press Enter to edit.

stack72 commented 7/26/2026, 11:14:26 PM

Hey @magistr — thanks for the detailed write-up, especially the grep verification that the colliding methods aren't in the pulled upstreams and the observation that only the first method in each extension's map collides (signature of double-registration). That narrowed the investigation significantly.

I spent a good chunk of time tracing through the extension loading paths and built a reproduction scenario to try to trigger the flap:

What I tested:

  • 191 total sources across 12 extensions (exceeding your 106 across 9)
  • Pulled the exact extensions from your report: @magistr/firecracker, @evrardjp/openbao-configurator, plus 9 others from @swamp/aws/* and @webframp/extension-maintenance
  • 9 local sources: 3 export const model + 6 export const extension targeting pulled types
  • Local extensions with transitive Zod dependencies via a shared _lib/ module

Variations tried (all passed consistently):

  • 20 consecutive runs (cold + warm starts)
  • 15 runs specifically with @magistr/firecracker and @evrardjp/openbao-configurator as targets
  • Deleting the catalog DB between sets to force cold starts
  • Deleting local bundle cache to force fresh bundling
  • Touching/modifying shared lib between runs to force fingerprint changes and rebundling
  • Parallel concurrent doctor invocations

None of these reproduced the alternating exit 1, 0, 1 pattern. All runs passed.

What I found in the code: The invalidateAll() → reconcile → buildIndex() flow does have an asymmetry: when reconcile finds 0 transitions (all sources already Indexed), it doesn't re-set the populated flags that invalidateAll() cleared, which pushes buildIndex into the cold path. But in my testing the cold path's processSecondaryExport never hit a collision — the model types were freshly registered with only their base methods.

Could you help narrow this down?

  1. Can you capture swamp doctor extensions --verbose --json from a failing run? The per-source detail and catalog state would show exactly which code path wrote the BundleBuildFailed.
  2. Does your repo have a deno.json or package.json at the root? Those can affect how Deno resolves imports during bundling, which could make the bundle output nondeterministic.
  3. Is the repo on a network filesystem, Docker volume, or case-insensitive mount?
  4. Does running swamp doctor extensions --repair once break the cycle permanently? If so, it points to corrupt catalog state carried forward from a prior swamp version rather than a live code path bug.

magistr commented 7/27/2026, 8:05:07 AM

Thanks for digging in that far — the invalidateAll() → reconcile → buildIndex() asymmetry looks right, and I think I can now give you the missing half of the repro. Three new facts first, then your four questions in order.

1. It is not nondeterministic — it strictly ALTERNATES

My original report said "1, 0, 1" and undersold it. Eight consecutive runs, unchanged tree, current build:

run 1: exit 1
run 2: exit 0
run 3: exit 1
run 4: exit 0
run 5: exit 1
run 6: exit 0
run 7: exit 1
run 8: exit 0

A clean 2-cycle, not a race. That matches your populated-flag finding exactly, and I think the loop closes like this:

  • a FAILING run leaves the catalog in a state the next run repairs — the PASSING run logs Extension catalog updated: 2 "entries" repaired, which is the same line --repair emits;
  • that repair leaves every source Indexed;
  • so the next run's reconcile finds 0 transitions, doesn't re-set the populated flags invalidateAll() cleared, takes the cold path, and collides;
  • fail → repair → all-Indexed → 0 transitions → cold path → fail.

If that is right, the flap is not timing-dependent at all and a repro needs only to enter the failing half once. A 20-run loop that never entered it would pass 20/20, which may be what happened on your side.

One possible differentiator: both of my colliding local sources are export const extension targeting a single-source pulled aggregate — @magistr/firecracker has sourceCount: 1, @evrardjp/openbao-configurator has sourceCount: 1. Your reproduction spread 6 extension sources across 11 extensions including @swamp/aws/* (many sources each). If the cold path re-registers a whole aggregate before merging secondary exports, a 1-source aggregate would be the degenerate case that double-registers.

2. --repair does NOT break the cycle

Answering your Q4 directly, because it rules out your "corrupt state from a prior version" theory:

swamp doctor extensions --repair -y
# Extension catalog updated: 2 "entries" repaired
# Repair APPLIED   (exit 0)

# then, immediately:
run 1: exit 1
run 2: exit 0
run 3: exit 1
run 4: exit 0
run 5: exit 1
run 6: exit 0

The repair applies cleanly and the very next run fails again, with the alternation resuming in phase. This is a live code path, not carried-forward corruption. Note also that --repair itself behaves as one "pass" step of the cycle — same repair line, same effect on the next run.

3. Still reproduces on a newer binary

Filed against 20260725.011748.0-sha.bb2cb3ef; the above is all on 20260727.000815.0-sha.db74fcac, unchanged.

Your four questions

Q1 — --verbose --json from a failing run. Captured, but it does not carry what you need, which is worth a separate small bug. On a failing run (overallStatus: "fail") the registry object is only:

{ "registry": "model", "status": "fail" }

No per-source detail, no error strings, and "orphanFiles": [] even though the human-readable run in the same phase reports 2 orphan bundle file(s). The text --verbose does have it:

extension .../extensions/models/marshland-myr.ts   BundleBuildFailed: Error: Method 'build_myr_rootfs' already exists on model type '@magistr/firecracker'      fp:b6adc7fb4106
extension .../extensions/models/mosse-admin.ts     BundleBuildFailed: Error: Method 'ensure_tenant_self_policy' already exists on model type '@evrardjp/openbao-configurator'  fp:e1a5cd8f3ce0
  0 orphan row(s), 2 orphan bundle file(s)
  @local/marshland   Indexed: 3 | BundleBuildFailed: 2
3 passed, 1 failed — OVERALL: FAIL

Both colliding names are still the FIRST method in their extension's map. Happy to attach the full JSON if useful, but the per-source array simply is not in it.

Q2 — root deno.json / package.json. No package.json, ever. There IS a deno.json now, but I added it after filing this issue, and the flap is identical on both sides of that change: the original 1,0,1 was observed with no deno.json in the repo at all, and the 8-run alternation above was observed with one present. So a root Deno config is not the trigger.

Q3 — filesystem. Local APFS (/dev/disk1s1), not a network mount, not a Docker volume. It IS case-insensitive — verified live (touch .casetest_ABC then ls .casetest_abc resolves). If bundle or catalog keys are ever compared case-sensitively while the filesystem folds case, that would be a plausible source of a double-registration.

Q4 — --repair. Covered above: it does not break the cycle.

Workaround still stands

swamp model type describe <type> --compact remains stable across runs and lists every contributed method, so asserting the property doctor proxies for — that the contributed methods resolve on their types — is reliable where doctor is not. I have that committed as a deploy precondition now: 11 extension-contributed methods across the two pulled types, plus all three local model types, anchored on the method separator rather than a bare substring.

stack72 commented 7/27/2026, 1:37:27 PM

@magistr — follow-up on the repro attempt. I rebuilt the scenario using your new details:

  • Pulled the exact @magistr/firecracker and @evrardjp/openbao-configurator extensions (both single-source aggregates, sourceCount: 1)
  • Created 2 local export const extension sources targeting those types with the exact colliding method names (build_myr_rootfs, ensure_tenant_self_policy, etc.)
  • Created 3 local export const model sources to match your 3+2 layout
  • Even manually seeded BundleBuildFailed in the catalog to try to enter the failing half of the cycle

All runs pass consistently — 8/8, every time. The reconcile repairs the seeded state on run 1 and the cold path's processSecondaryExport never hits a collision after that.

One thing I want to confirm: is this strictly deterministic for you? You showed a clean 8-run alternation — does that happen every single time, from any starting state, on any terminal session? Or is there any variation (e.g., it only starts after a specific command, or it clears after a reboot, or it depends on which directory you run from)?

If it's truly deterministic and locked to your repo, the fastest path forward would be for you to share a sanitized copy of the repo (just the .swamp/ directory, extensions/, and .swamp.yaml — no secrets or application code needed). I can reproduce against the exact catalog state and extension structure rather than continuing to approximate it. Happy to work with a tarball, a throwaway git repo, whatever's easiest.

stack72 commented 7/29/2026, 4:55:58 PM

@magistr ping to put this back on your radar

magistr commented 7/29/2026, 7:42:40 PM

@stack72 — no need for my repo, here's a self-contained repro. It reproduces on the first run, which I think is why your 20-run loop never saw it.

Run this. It builds a throwaway repo in /tmp, pulls two extensions, writes five trivial local sources, and runs doctor eight times. ~2 minutes.

#!/usr/bin/env bash
set -euo pipefail
DIR="${1:-/tmp/swamp-1419-repro}"
rm -rf "$DIR"; mkdir -p "$DIR/extensions/models"; cd "$DIR"

swamp repo init >/dev/null
swamp extension pull @magistr/firecracker           >/dev/null
swamp extension pull @evrardjp/openbao-configurator >/dev/null

# Two `export const extension` sources. Method names are invented, so a reported
# collision cannot be a real clash with upstream. Each declares TWO methods on
# purpose — only the first is ever reported.
for pair in "alpha:@magistr/firecracker:ext-firecracker" \
            "beta:@evrardjp/openbao-configurator:ext-openbao"; do
  tag="${pair%%:*}"; rest="${pair#*:}"; type="${rest%%:*}"; file="${rest##*:}"
  cat > "extensions/models/$file.ts" <<EOF
import { z } from "npm:zod@4";
const args = z.object({ note: z.string().default("repro") });
export const extension = {
  type: "$type",
  methods: [{
    repro_${tag}_first_method: {
      description: "Lab #1419 repro. Does nothing.",
      arguments: args,
      execute: (a: z.infer<typeof args>) => ({ ok: true, note: a.note }),
    },
    repro_${tag}_second_method: {
      description: "Second method — never reported as colliding.",
      arguments: args,
      execute: (a: z.infer<typeof args>) => ({ ok: true, note: a.note }),
    },
  }],
};
EOF
done

# Three plain `export const model` sources. These index cleanly on every run.
for n in one two three; do
  cat > "extensions/models/model-$n.ts" <<EOF
import { z } from "npm:zod@4";
const G = z.object({ label: z.string().default("$n") });
const A = z.object({ value: z.string().default("x") });
export const model = {
  type: "@repro/model-$n",
  version: "2026.07.29.1",
  globalArguments: G,
  resources: {
    state: { description: "Trivial.", schema: z.record(z.string(), z.unknown()),
             lifetime: "1h", garbageCollection: 5 },
  },
  methods: {
    noop: { description: "Does nothing.", arguments: A,
            execute: (a: z.infer<typeof A>) => ({ ok: true, value: a.value }) },
  },
};
EOF
done

swamp extension source add extensions/models >/dev/null

for i in 1 2 3 4 5 6 7 8; do
  swamp doctor extensions >/dev/null 2>&1 && echo "run $i: exit 0" || echo "run $i: exit 1"
done

You should see:

run 1: exit 1
run 2: exit 0
run 3: exit 1
run 4: exit 0
run 5: exit 1
run 6: exit 0
run 7: exit 1
run 8: exit 0

Still reproducing three builds after I filed this:

swamp 20260728.192016.0-sha.f2a1f6b9
deno 2.7.13, macOS x86_64

A failing run reports:

extensions/models/ext-firecracker.ts  BundleBuildFailed:
  Error: Method 'repro_alpha_first_method' already exists on model type '@magistr/firecracker'
extensions/models/ext-openbao.ts      BundleBuildFailed:
  Error: Method 'repro_beta_first_method' already exists on model type '@evrardjp/openbao-configurator'

  @local/... 5 source(s)   Indexed: 3 | BundleBuildFailed: 2
  0 orphan row(s), 2 orphan bundle file(s)
3 passed, 1 failed — OVERALL: FAIL

What it shows

1. It fails on run 1, from a cold catalog. No prior state is needed to enter the failing half — a fresh repo starts there. That's probably why your 20/20 passed: "always passes" and "always alternates" are both stable states of the same machine, and which one you land in is decided before the first run. So my earlier framing was incomplete — the repair → 0-transitions → cold-path loop explains what sustains the flap, not what enters it.

2. Only the first key in each methods map is ever reported. Both extensions declare two methods; repro_alpha_second_method and repro_beta_second_method are never named. Consistent with the aggregate being re-registered and the merge re-adding from the top of the map.

3. My sourceCount: 1 hypothesis was wrong — please ignore it. Add a third export const extension targeting @swamp/digitalocean/droplet (47 sources) to the repo above and it flaps identically, in the same runs, beside the two single-source ones:

ext-digitalocean.ts BundleBuildFailed: Method 'repro_gamma_first_method' already exists on model type '@swamp/digitalocean/droplet'
ext-firecracker.ts  BundleBuildFailed: Method 'repro_alpha_first_method' already exists on model type '@magistr/firecracker'
ext-openbao.ts      BundleBuildFailed: Method 'repro_beta_first_method'  already exists on model type '@evrardjp/openbao-configurator'

So spreading across @swamp/aws/* isn't what stopped your reproduction — no need to keep chasing single-source aggregates.

4. It's deterministic and not session- or cwd-bound. Fifteen runs against my real repo: 1,0,1,0,1,0,1,0,1,0,1 from the repo, then 0,1,0,1 from /tmp with --repo-dir. The cycle continued without breaking parity, so the state is in the repo catalog. I can predict which run fails and be right every time.

One aside

--verbose --json from a failing run won't help you — I captured one and it carries none of the per-source detail:

{ "overallStatus": "fail",
  "registries": { "model": { "registry": "model", "status": "fail" } },
  "loaderErrors": {}, "warnings": [], "recentTransitions": [] }

loaderErrors, warnings and recentTransitions are all empty, and the registry entry has only registry and status. The BundleBuildFailed messages exist only in the human-readable output. Happy to file that separately if it's useful.

stack72 commented 7/29/2026, 8:59:34 PM

@magistr — we found it. The repro script nailed it, thank you.

The key line is swamp extension source add extensions/models. That directory is already the default models scan path — resolveModelsDir() returns "extensions/models" and every buildIndex() call uses it as the primary directory. Adding it to .swamp-sources.yaml causes it to also appear in additionalDirs, so load() iterates [dir, ...additionalSet] and discovers every file twice.

For export const model files this is harmless — skipAlreadyRegistered silently drops the duplicate registration. For export const extension files, processSecondaryExport calls modelRegistry.extend() twice with the same methods. First call succeeds, second call throws "Method already exists on model type."

The alternation is exactly the populated-flag asymmetry you identified: the BundleBuildFailed triggers a repair on the next run, repair sets populated flags, warm path runs (which has isExtensionAttached dedup), passes. Next run clears populated, cold path runs, double-processes, fails again.

Fix: remove the source mount — your local extensions are already scanned by default:

swamp extension source remove extensions/models

After that, swamp doctor extensions should pass consistently. The source add command is for mounting external directories (monorepo siblings, shared libraries) — the local extensions/ tree doesn't need it.

stack72 commented 7/29/2026, 10:33:37 PM

Added a guard in https://github.com/swamp-club/swamp/pull/2011

Sign in to post a ripple.