Relationships
#4536 API runs out of Mongo connections under bursts: extension push timeouts, membership failures and UAT 500s
Opened by swamp_lord · 11/9/2026· Shipped 11/9/2026
Summary
The swamp-club API runs out of MongoDB connections under ordinary bursts of
load. Requests then wait up to waitQueueTimeoutMS (5s) for a connection and
fail with "Timed out while checking out a connection from connection pool".
The median Mongo call stays at about 6ms throughout; only the tail stalls, so
it looks like random slowness rather than an outage.
The visible effects:
swamp extension pushfails withPOST /api/v1/extensions/confirm timed out after 15s(the CLI's timeout). The server finished the confirm about 1.5s later, so the version is published, but the publisher reports a failure.Could not verify membership — please check your network connectionon push. The network was fine; the membership check waited on the pool.- UAT failures:
Extension API error (HTTP 500): Internal server errorwhile auto-resolving@swamp/s3-datastoreand@swamp/gcs-datastore. - platform-swamp's own release pipeline failed with an HTTP 500 on
GET /api/v1/extensions/@swamp/ci-uatat 2026-11-09 10:29 UTC, during quiet traffic.
Related: #759 (pool raised from 10 to 50), #1088 (an earlier Mongo p95 degradation), #4050 (streak projector full scan every minute).
Evidence (Axiom swamp_club_prod, OTel spans)
| When (UTC) | Request p99 | Mongo p99 | Pool-checkout timeouts | What broke |
|---|---|---|---|---|
| 2026-11-08 02:17–02:28 | up to 21s | 3–7s (max 27s) | on extension.findByName (35), extension.getVersionByName (21), collective-api-token.recordLastUsed (19), collective.findById (14) and the watchers, 02:00–02:45 |
Bulk @swamp/aws/* publish: @swamp/aws/codedeploy confirm took 16.46s (checkMembership 6.4s, then findByName 9.4s); 17 swamp-uat jobs failed with HTTP 500 between 02:18 and 02:59 |
| 2026-11-08 23:28–23:41 | up to 17.9s | up to 3.9s | 67 in 11 minutes | checkMembership 2–13s on almost every call, several ending at about 5.1–5.4s with no outcome recorded (consistent with the 5s wait-queue timeout); @swamp/aws/apprunner push failed membership; swamp-uat run 42533389130 failed with HTTP 500 at 23:34:21 |
| 2026-11-09 10:20–10:35 | — | — | 3 | One release pipeline lookup failed with HTTP 500 |
During both incidents, 20–30 different Mongo operations went over 1s at
once while the median stayed at about 6ms, including trivial indexed reads
(collective.findById, operative.findById). That points to the shared pool,
not one bad query.
From the cluster (DOKS swamp-club-prod, swamp-club/swamp-club-api, as of 2026-11-09):
- The autoscaler is pinned at its maximum of 10 replicas. It scales on memory, and pods use about 930Mi of a 1Gi request (86%), while CPU averages 42% of request (about 0.1 core of a 2-core limit). So there are never spare replicas, or spare pools, when a burst arrives.
- No pod restarted or was OOMKilled during either incident.
Why the pool runs out
lib/db.ts: maxPoolSize: 50, minPoolSize: 5, waitQueueTimeoutMS: 5000,
socketTimeoutMS: 20000, on one MongoClient per pod shared by the request
path and every background loop: lib/db.ts is the only client, and
main.ts starts 14 watcher and queue loops in the API process. Even at baseline, about 10 Mongo calls a
minute take over 1s, and some background reads hold a connection far longer:
tier-announcement.findAllOrdinals (up to 27s), serve-link.claimPendingEmails
(up to 12.5s), board-freshness.current (up to 10.5s). A burst of publishes
(each one runs several sequential Mongo calls) on top of those fills all 50
connections, and everything else queues.
Fix for good
- Separate the background work from requests. Give the watchers, queues and projectors (
watcher.*,queue.*,app.score.*, streaks) their ownMongoClientwith a small pool (for example 10), or move them out of the API pods. A slow sweep must never take a request's connection. - Put a time limit on every query. Set
maxTimeMS(driver-leveltimeoutMS, or per query) well undersocketTimeoutMS, so a slow query frees its connection instead of holding it for up to 20s. - Fix the slow queries listed above, plus
extension.findByNameandextension.getVersionByName(the publish hot path). Check Atlas Performance Advisor and the profiler for missing indexes and collection scans. #4050 is one of these. - Size the pool against Atlas. Raise
maxPoolSizefor the request client (for example 100) only after checking the Atlas tier's connection limit across all replicas, the background clients, telemetry and the Discord bot. KeepwaitQueueTimeoutMSbelow the CLI's timeouts, and make the error a 503 withRetry-After, not a 500, so clients can tell "busy" from "broken". - Fix the autoscaling. Raise the memory request (about 1.5Gi) so memory stops pinning the autoscaler, scale on CPU and request latency, and review the maximum of 10.
- Make it observable. Export the driver's connection-pool (CMAP) events as metrics or span attributes: checked-out connections, wait-queue length, checkout wait time and checkout failures. Add the
k8s.pod.nameresource attribute so latency can be split by pod. Also fix trace context propagation: unrelated requests currently share one trace ID (one trace held 395 spans across many extensions). - Client side (swamp CLI, to file separately): retry
extensions/confirmand the membership check with backoff, treating "already published" as success, and stop reporting server timeouts and 5xx as "check your network connection".
Axiom monitors to add
The thresholds come from the last 7 days (2,017 five-minute bins). Route the
notifications to ops (https://ops.swamp-club.com/hooks/axiom-alerts), so they
open an incident there and can be checked with swamp-club-prod-debug investigate.
1. Pool checkout timeouts (page). Over the 7 days: 17 five-minute bins with any, 11 with ≥5, max 91.
['swamp_club_prod']
| where ['service.name'] == 'swamp-club'
| where ['status.message'] contains 'checking out a connection from connection pool'
| summarize n = count() by bin(_time, 5m)Threshold monitor: n >= 5 in 5 minutes. A warning-level copy at n >= 1 catches the start of an incident.
2. Extension API errors (page). /api/v1/extensions* served 1.26M requests with 68 5xx in 7 days: 9 bins with any, 5 with ≥3, max 18.
['swamp_club_prod']
| where ['service.name'] == 'swamp-club' and kind == 'server'
| extend target = tostring(['attributes.custom']['http.target']),
code = toint(['attributes.custom']['http.status_code'])
| where target startswith '/api/v1/extensions' and code >= 500
| summarize n = count() by bin(_time, 5m)Threshold: n >= 3 in 5 minutes.
3. Extension API close to the CLI timeout (warn). 13 bins in 7 days had a request over 10s; the CLI gives up at 15s.
['swamp_club_prod']
| where ['service.name'] == 'swamp-club' and kind == 'server'
| where tostring(['attributes.custom']['http.target']) startswith '/api/v1/extensions'
| extend ms = totimespan(duration) / 1ms
| summarize slow = countif(ms > 10000) by bin(_time, 5m)Threshold: slow >= 1. The confirm-push span (app.extension.confirm-push.confirm-push) can have its own copy.
4. Mongo p99 (warn). The 5-minute p99 has a median of 74ms but goes over 2s in 243 bins out of 2,017, because of the slow background reads above; it went over 3s only 7 times. Use 3s until the background work is split off (fix 1), then lower it to about 1s.
['swamp_club_prod']
| where ['service.name'] == 'swamp-club' and name startswith 'infrastructure.mongo.'
| extend ms = totimespan(duration) / 1ms
| summarize p99 = percentile(ms, 99) by bin(_time, 5m)Threshold: p99 > 3000 for 2 consecutive bins.
5. Membership check latency (warn). It is on every push and sat at 2–13s during the second incident.
['swamp_club_prod']
| where ['service.name'] == 'swamp-club'
| where name == 'infrastructure.mongo.namespace-ownership.checkMembership'
| extend ms = totimespan(duration) / 1ms
| summarize p95 = percentile(ms, 95) by bin(_time, 5m)Threshold: p95 > 2000.
The autoscaler at its maximum isn't visible in Axiom; it's a cluster alert (a
kube-state-metrics rule, or a scheduled prod-debug workload_status on ops).
How this was found
On ops, with swamp-club-prod-debug (@swamp/platform-ops/prod-debug):
investigate over the incident windows, axiom_query for the baselines,
workload_status for the autoscaler and pods, and ci_failures and ci_run
across GitHub and Forgejo for the CI failures. The results are kept on ops as
investigation/confirm-timeout-20261108, investigation/membership-check-20261108
and axiom/baseline-*-7d.
Shipped
Click a lifecycle step above to view its details.
system commented 11/9/2026, 10:35:25 AM
Classified automatically when this issue was filed.
- Source: Swamp Club
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.