#1317 [sc] quest-grant Mongo ops (recentHeartbeat/signalCounts/countIssueShipped) regressed to ~5s timeouts — missing index

Summary

The quest-grant MongoDB operations have regressed into hard ~5s timeouts, firing the [sc] Mongo latency high (p95) alert on 2026-07-21 ~12:00 UTC (overall infrastructure.mongo.* p95 = 1.27s vs 500ms threshold). This is a slow-burn regression tied to collection growth since the feature rolled out, not the previously-fixed Lab #1071 subscription N+1.

Offending operations (last 30m, prod spans swamp_club_prod)

Every other Mongo op is healthy (10–100 ms). These three are dragging the fleet p95 over threshold on their own.

Timeline (7d p95 for quest-grant.*)

07-16 18:00     86 ms   feature fully rolled out, healthy
07-17/18      ~100–260 ms
07-19 06:00   1175 ms   degradation begins
07-19 12:00   2129 ms
07-20/21      ~900 ms–1.2 s (chronic)
07-21 12:00   5087 ms   hit the ~5s ceiling -> alert fired

Diagnosis (high confidence): missing/insufficient index

• Bimodal latency — recentHeartbeat p50 is only 109 ms but p95 ~5 s. Most calls hit cache; the tail does a full COLLSCAN.
• Degrades with data growth — steadily worse since rollout, the signature of an unindexed scan over a growing collection (heartbeats / signals).
• Pinned at ~5 s (max 5.18–5.24s) — the scan runs into maxTimeMS / socket timeout.
• signalCounts is worse at the median (p50 730 ms), suggesting an unindexed aggregation/count.

Suggested fix

Add indexes backing these queries in the quest-grant data layer:

• recentHeartbeat — likely on the heartbeat collection's (grantee/user id, timestamp) fields.
• signalCounts / countIssueShipped — likely on the signals collection's grouping key.

Confirm exact query shapes via the span attributes / quest-grant repo code.

Environment

• Service: swamp-club (namespace swamp-club-platform), Node/OpenTelemetry.
• Dataset: swamp_club_prod (Axiom, spans-only).
• Detected via alert [sc] Mongo latency high (p95) on 2026-07-21.