Skip to main content
← Back to list
01Issue
BugShippedSwamp Club
Assigneeskeeb

Relationships

#1496 Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

Opened by keeb · 8/2/2026· Shipped 8/2/2026

Problem

lib/app/score-read-fallback.ts:58 declares a single process-wide CircuitBreaker (threshold 3, cooldown 30s):

const breaker = new CircuitBreaker({ threshold: 3, cooldownMs: 30_000 });

It is shared by both withBoardRead and withScoreReadFallback.

Consequence: three consecutive leaderboard board timeouts open the breaker for every score read in the process — profile score reads and tier resolution included — for 30 seconds. A burst of leaderboard traffic therefore degrades surfaces that have nothing to do with the leaderboard: an operative's profile score blanks, and tier-colored name rendering loses its tier data, because the board next door was slow.

Why the leaderboard is the likely tripwire

Measured 2026-08-01 (local ClickHouse system.query_log, dataset 17.35M events; prod swamp.events is 74.75M rows / 6.72 GiB):

  • One /leaderboard render fires 6 ClickHouse queries, ~2.36M rows read, ~660 MiB peak memory. Four of them are concurrent via Promise.all in lib/app/leaderboard-view.ts:100-105.
  • A 12-concurrent-viewer test: 48 queries, 7.66 GiB summed memory, page latency 0.12s -> 0.99s.
  • Every score read runs under a 5s max_execution_time (lib/infrastructure/clickhouse/clickhouse-score-reads.ts:462, SCORE_READ_CEILING_S) with a 7s client abort.

So the leaderboard fan-out is by far the most likely thing to hit the ceiling three times in a row — which means the noisiest neighbour is also the one that takes everything else down with it.

The tradeoff is deliberate, and needs a decision — not a blind change

The comment at lib/app/score-read-fallback.ts:38-57 argues for the shared breaker: every score-read seam fronts the same ClickHouse, so one ClickHouse is one health signal, and under a genuine full outage the sharing is what stops a four-board page from stacking timeouts into a 504. That reasoning is sound for a total outage.

It is not sound for the case actually observed here: ClickHouse is healthy, and only the heaviest query class is exceeding the ceiling. In that regime the shared signal is wrong — it reports "ClickHouse is down" when the truth is "the streak board is too expensive."

Proposal

Separate breakers per read class:

  • board reads (withBoardRead) — the 4-way fan-out
  • profile/tier reads (withScoreReadFallback)

At minimum, give the board path its own instance so board pressure cannot blank profile scores. If the full-outage protection is worth keeping, the two can share an outer signal (e.g. a breaker that only opens on connection-level failures) while per-class breakers handle timeouts.

Filing this so the tradeoff gets made explicitly rather than being rediscovered during an incident. The existing comment should be updated with whichever way the decision goes.

Notes

  • The boot probe in repos.ts bypasses the breaker by calling the primary directly, so it is unaffected either way.
  • There is no Mongo fallback anymore — a tripped read throws (profile/tier) or degrades to an "unavailable" board — so an over-eager breaker is directly user-visible.
  • One of three issues from the same 2026-08-01 /leaderboard investigation; the other two are performance work on the queries that make this breaker likely to trip.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 11 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

8/2/2026, 7:53:01 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb8/2/2026, 6:26:22 PM
Editable. Press Enter to edit.

keeb commented 8/2/2026, 2:12:32 AM

Siblings from the same 2026-08-01 /leaderboard investigation: #1495 (materialize active-days-by-id for the streak board) and #1497 (ghost-gate dictionary + eventCounts pushdown). Both reduce the pressure that makes this breaker trip, but neither removes the shared-signal tradeoff — that still needs an explicit decision.

Sign in to post a ripple.