Skip to main content
← Back to list
01Issue
BugClosedPlatformPublic
AssigneesNone

Relationships

#4872 Extension scoring blocks the API event loop for over a minute on large extensions, causing 503s on the scoring pod

Opened by swamp_lord · 11/18/2026

What happens

When swamp-club scores a newly published extension, the analysis runs on the API process's main thread and blocks its event loop for most of the run. For a large extension, the API pod stops serving requests in good time for over a minute. Requests that land on that pod wait more than 5s for a Mongo connection and fail with Timed out while checking out a connection from connection pool, which the API returns as HTTP 503.

Seen on 2026-11-18 at 13:12–13:16 UTC, during a bulk @swamp/aws/* publish. Three API pods were scoring @swamp/aws/ec2, @swamp/aws/lambda and @swamp/aws/sagemaker, and those three pods, and only those, returned 87 503s across /api/whoami and /api/v1/extensions/*. A concurrent CLI run failed with:

Error: Registry check "authentication" could not run: registry did not answer: Whoami request failed (HTTP 503)

The request pool (150 connections) was not the limit. The driver's checkout and its timeout timer run on the same event loop the analysis is blocking.

Reproduction

The repro runs extractContentMetadata and DenoDocExtensionTarballAnalyzer.analyze on @swamp/aws/ec2@2026.11.18.1 (26 MB archive, 142 MB unpacked, 126 models), with a 20ms setInterval probe for event-loop lag. It compares the revision deployed during the incident with the parent of d3f31a32, on an M-series laptop. The runs pinned to efficiency cores (taskpolicy -c background) stand in for slower production vCPUs.

revision cores total stalls >100ms lag p90 / p99 / max
430309d0 (before d3f31a32) fast 2.9s 1 1 / 12 / 561 ms
8601a8b3 (deployed) fast 12.1s 9 80 / 112 / 558 ms
430309d0 efficiency 4.0s 1 2 / 38 / 1092 ms
8601a8b3 efficiency 23.8s 79 172 / 283 / 1096 ms

In production the same batch took 81.6s, on shared vCPUs while the pod was also serving a request burst.

A CPU profile of the deployed revision on efficiency cores shows scanCode taking about 74% of the CPU time. Most of that is markdownImports, which runs a full @babel/parser parse and AST walk (importedSpecifiers → importedSpecifiersAt → importedSpecifiersIn) over every code file, only to find which .md files are imported. Nothing caps it. writtenSpecifiers, the dependency extraction, re-parses only the files within MAX_SOURCE_FILES / MAX_SOURCE_BYTES (500 files / 50 MB) and costs much less. The deno doc steps already run as subprocesses and cost little on the main thread. The work is synchronous per file, and the loop yields only on the file read between files.

Before d3f31a32 (2026-10-26, swamp-club#3977), dependency extraction was a regex scan (extractDependencySpecifiers), and the same archive held the loop for under 0.6s in total.

The per-event timeout in processOne (Promise.race) can't interrupt synchronous parsing, so it doesn't bound the stall.

Expected

Scoring an extension of any size must not stall request handling on the pod that runs it.

Suggested fix

  1. Run the analysis (extractContentMetadata, analyzer.analyze) off the main thread, in a Worker or subprocess, and terminate it on timeout.
  2. Make import discovery cheap in @swamp-club/extension-scoring (packages/extension-scoring in swamp): parse each file at most once, and cap or short-circuit markdownImports (most files have no .md import).
  3. Add a span per scoring phase, and an event-loop lag metric.
02Bog Flow
✓OPEN○TRIAGED○IN PROGRESS◉CLOSED

Closed

11/18/2026, 8:10:49 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

swamp_lord commented 11/18/2026, 4:22:01 PM

Correction to the triage record's regression numbers. The first local run (88s, 77.5s of event-loop stall) was an anomaly I couldn't reproduce. It was probably memory pressure on the laptop: that profile showed 16s of GC, and the machine had 6 GB in swap. Repeated runs under identical conditions:

  • before d3f31a32 (430309d0): 2.9s, 1 stall over 100ms (fast cores); 4.0s, 1 stall (efficiency cores)
  • deployed 8601a8b3: 12.1s, 9 stalls, p99 lag 112ms (fast cores); 23.8s, 79 stalls, p90/p99 172/283ms (efficiency cores) The main cost is markdownImports doing an uncapped Babel parse and AST walk over every code file. Writtenspecifiers' re-parse is capped and cheaper. The issue body has these numbers. The verdict (regression from d3f31a32) stands.

swamp_lord commented 11/18/2026, 6:40:26 PM

Verified end to end locally (Mongo and LocalStack, deno task start, real swamp CLI), main 526bde0b against this branch, back to back. A real push of a 126-model extension took the server's extraction fallback, then the watcher scored it, with 4 loops probing /api/whoami and the extension endpoint throughout.

  • API during push and scoring: main p99 1015ms, max 1374ms, 11 requests over 1s. Branch p99 19/17ms, max 31/40ms, 0 over 1s (two runs, all 200s).
  • Server push and confirm: main 21/831ms, branch 18/872ms (about 40ms of child start-up).
  • An old-client push (no contentMetadata field) gives confirm 201 with the same response on both.
  • Stored version documents (contentMetadata, 126 models) and scorecards (25 fields) are identical between the builds. Production image: both worker modes run from /app with results identical to in-process; child start-up is 72ms for extract and 344ms for score.

swamp_lord commented 11/18/2026, 7:17:45 PM

Shipped: swamp-club PR #1535 merged (94e435658) and deployed to DOKS at 19:16 UTC. swamp-club-api is 10/10 on the new ReplicaSet with 0 restarts. Pending: confirming the first production scoring run through the child process (worker spans with exit code 0 and fallback=false, and flat per-pod p99). No extension has been scored since the rollout yet. Follow-up: #4874 (parse cost in the extension-scoring package).

Sign in to post a ripple.