Relationships
#4895 swamp behavior list re-scans the extension model-definition tree once per distinct type (O(types x extension-definition-files) file opens)
Opened by kneel · 11/19/2026· Shipped 11/19/2026
Summary
On a repo with many distinct extension types, swamp behavior list spends
~20-26s wall (~13-15s user-mode CPU) almost entirely re-reading and re-parsing
the same extension-namespaced model-definition YAML files. A single run issues
one swamp.model.get (type resolution) per distinct type -- 157 distinct type:
values in the test repo, matched 1:1 by 157 swamp.model.get spans -- and each
type resolution re-scans the full extension model-definition source tree. Net
effect: the ~807 .yaml files under models/ are opened 103,563 times in one
invocation, with each extension-namespaced definition re-opened ~157x.
The cost is O(distinct-types x extension-definition-files). It is NOT quadratic
in repo size and does NOT grow with the number of behaviors (adding behaviors of
an existing type is free). It is CPU-bound on YAML parsing, not datastore or
network I/O: the same run opens the local SQLite catalog _catalog.db 6 times
and the local datastore data cache (149,982 files) 0 times.
Environment
- swamp
20261114.210101.0-sha.537e03d1(please verify against head; may already be addressed). - Datastore: a third-party blob-storage datastore provider with a local SQLite
catalog
_catalog.dbatcatalog_meta.schema_version = 9. The hot path is in the local filesystem type-resolution layer, above the provider, so this is expected to reproduce on any provider (not directly confirmed on a second provider -- see Open questions). - Repo scale: 157 distinct
type:values (156 extension-namespaced plus the built-incommand/shell), 807 git-tracked.yamlfiles undermodels/, roughly 800 behaviors. - Linux x86_64.
Impact
swamp behavior listtakes ~20-26s wall / ~19-22s CPU; the time is spent re-parsing extension-namespaced YAML definitions.- 0 bytes network, ~6 catalog opens, 0 data-cache opens, so the latency is pure redundant definition parsing.
- It is the everyday "what do I have" command, so the cost is felt constantly and grows as more distinct extension types are installed/used.
Evidence (syscall + span instrumentation)
Instrumented one swamp behavior list (telemetry off) with
strace -f -e trace=openat, and separately with OTEL_TRACES_EXPORTER=console.
Open counts (successful openat), bucketed:
models/source tree (all paths): 103,563- ... distinct paths: 1,001 (807
.yaml+ 194 directories) - ... re-open factor, extension-namespaced
.yaml: 155-159x - ... re-open factor, built-in
command/shell.yaml: ~1x (read once) - local SQLite catalog
_catalog.db: 6 - local datastore data cache (149,982 files): 0
Span tree (console exporter), same command:
swamp.model.searchx1swamp.model.getx157 (each carriesattributes.model.id_or_name)swamp.datastore.extension.type_load/ teardown spans
The join that pins the mechanism: the repo has exactly 157 distinct type:
values, and there are exactly 157 swamp.model.get spans -- one type resolution
per distinct type. The per-file re-open count on extension definitions (157-159x)
equals that type count. Built-in command/shell definitions are opened ~once, so
the re-scan is scoped to extension type resolution, not a reload of the entire
definition set.
Timing (clean, untraced, telemetry off), varies run to run, always CPU-bound:
real ~19-26s user ~13-15s sys ~6sReproduction
The amplifier is the number of distinct extension types, not the number of behaviors.
- A repo with many distinct extension types (each from an extension namespace,
e.g.
@collective/pkg/type), with definition files undermodels/. time swamp behavior list >/dev/null-> multi-second CPU dominated by YAML parse.- Show the re-scan:
strace -f -e trace=openat -o /tmp/ol.log swamp behavior list >/dev/null- total extension-def opens:
grep -c '/models/@' /tmp/ol.log - distinct
.yaml:grep -oE '"[^"]*/models/[^"]+\.yaml"' /tmp/ol.log | sort -u | wc -l - distinct types resolved: count of
swamp.model.getspans (equals the distincttype:count) The ratio (total / distinct) is approximately the distinct-type count, demonstrating the per-type re-scan.
- total extension-def opens:
Independent mock-repo control: in a fresh repo with N trivial single-type
behaviors, models/ opens are strictly linear (opens ~= N, each file opened
once) and wall time is flat at ~0.6s for N=10..200 -- confirming the cost does
NOT scale with behavior count. The amplification requires many distinct
extension types, not many behaviors.
Expected
Type resolution should load/cache each distinct extension type's definitions once
per invocation, rather than re-scanning the extension model-definition tree on
every per-type model.get / type_load. That turns the work from
O(types x extension-definition-files) into O(extension-definition-files),
bringing behavior list from tens of seconds to roughly one definition-tree
load.
What this is NOT (ruled out by the evidence)
- Not a datastore data-cache walk (0 opens of the data tree).
- Not datastore wire / provider latency (0 network; 6 catalog opens).
- Not addressable by
swamp data gc(gc removes expired data versions, whichlistnever reads). - Not quadratic in repo size, and not driven by behavior count or by the number of configured sources (a sources-count sweep plateaued at ~1.9x re-read, far from 157x).
- The local catalog's write-ahead log being large is a separate, secondary concern (only 6 catalog opens here); noted below, not the cause of this ticket.
Open questions for maintainers
- Is this already addressed on head? Reported build
20261114.210101.0-sha.537e03d1. - Does any in-flight datastore-engine work touch the extension
type_load/ per-typemodel.getdefinition-scan path, or only datastore I/O? If only the latter, this would survive that work -- the hot path is extension type resolution over local source files, not datastore reads. - Is the per-type re-scan intended (isolation between type loads), or an un-memoized hot path safe to cache within one invocation?
- Secondary: should the local catalog's WAL be checkpointed/bounded? A large WAL relative to the main DB suggests autocheckpoint is being starved.
Shipped
Click a lifecycle step above to view its details.
zack commented 11/19/2026, 5:30:33 PM
Thanks @kneel for reporting this! We shipped: behavior search/list in JSON output mode (--json or a non-TTY stdin) adds methods and globalArgumentsSchema per distinct type by running a full modelGet on that type's first definition by name. Each name lookup re-walks models/ until it matches, so the command costs O(types x definition files): at 10000 definitions and 200 types, 1005400 YAML opens and about 19.5s. The fix resolves the detail by type straight from the model registry, which needs no definition, so each definition file is read once by the listing walk (10000 opens, under 1s, identical output). The JSON renderer takes a type-keyed fetcher, the CLI always supplies both fetchers and the renderer picks by mode, and integration tests pin output parity with behavior get, the read count, the duplicate-name case, unregistered types and the interactive preview.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.