Skip to main content
← Back to list
01Issue
BugShippedExtensionsPublic
Assigneeszack

Relationships

#4874 extension-scoring: markdownImports parses every code file with Babel uncapped, and writtenSpecifiers parses them again

Opened by swamp_lord · 11/18/2026· Shipped 11/18/2026

What happens

@swamp-club/extension-scoring (packages/extension-scoring) does most of its archive-scan CPU work in markdownImports. It runs a full @babel/parser parse and AST walk (importedSpecifiers → importedSpecifiersAt → importedSpecifiersIn) over every code file, only to find which relative .md files are imported. Nothing caps the work. writtenSpecifiers then parses the same files again for dependency extraction (capped at 500 files / 50 MB). So a file is parsed twice, and the uncapped pass is the expensive one.

On @swamp/aws/ec2@2026.11.18.1 (126 models, 142 MB unpacked), a CPU profile of swamp-club's scorecard analysis shows scanCode at about 74% of CPU time. markdownImports accounts for about 85% of that, and GC on the large ASTs adds a lot more. The registry's scan took about 12s on a fast laptop core and 24s on an efficiency core, against about 3s before swamp-club switched to the package (swamp-club#3977). The same cost applies wherever the package runs, including swamp extension quality.

swamp-club is moving this work off its server's event loop (swamp-club lab #4872), which removes the production impact there. The CPU cost remains, and so does the CLI's.

Suggested fix

  1. Parse each module once and derive both the Markdown imports and the written specifiers from the same AST.
  2. Make markdownImports cheap. Its only output is relative specifiers ending in .md, so the walk can visit only import and export declarations and dynamic import() with a string argument, instead of the whole AST. A lexer-level scan might replace the parse altogether.
  3. Add a performance vector to the package's tests, for example 40 generated modules, so a regression shows up.

The result must stay identical. Run swamp-club's tests/integration/extension_scoring_vectors_test.ts and the package's own vectors.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 2 MORETRIAGE+ 8 MOREREVIEW+ 10 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/18/2026, 9:04:53 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/18/2026, 7:50:16 PM
Editable. Press Enter to edit.

system commented 11/18/2026, 6:14:12 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

swamp_lord commented 11/18/2026, 8:08:22 PM

Parked: no production impact left. Since swamp-club #4872 (PR #1535), registry scoring runs in a child process off the API's event loop, so this is now CPU efficiency only (registry scoring time, cold CLI quality and push). Notes for whoever picks it up, measured on @swamp/aws/ec2@2026.11.18.1 (253 audited code files, 142 MB) on a fast core:

  • Babel parse is about 3.4s per pass and the walk about 1.2s. markdownImports plus writtenSpecifiers parse every file twice (9.2-12.3s). The CLI parses the same files more times still: push markdownImports, the dependency extractor, and scanImportSites from three call sites. A cold swamp extension quality takes about 29s.
  • The exact lever is parse-once. A prototype took 9.2s to about 4.6s with identical output. Options: an explicit API (moduleImports plus markdownImportsIn/writtenSpecifiersIn, keeping today's functions as wrappers), and/or a bounded memo in importedSpecifiersAt so every CLI caller benefits.
  • Measured and dropped: a top-level fast path (exact, but bundles mention import/export everywhere, so no gain), attachComment:false (about 1%), and a for-in walk (slower).
  • This is not a regression to revert: df3636764 swapped the text scanner for Babel deliberately, for correctness.

Sign in to post a ripple.