Relationships
#1573 Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)
Opened by keeb · 8/10/2026· Shipped 8/10/2026
Phase 1 of #1572. No schema change. Highest value per unit of risk in the epic.
Problem
/leaderboard has no result cache and no request coalescing. Every concurrent viewer pays a full recompute of the entire owner population to see the same 25 rows.
Measured on the dev dataset (17.3M events), one board query, varying concurrency:
| Concurrent viewers | avg latency | peak mem/query | summed mem |
|---|---|---|---|
| 1 | 40 ms | 143 MiB | 143 MiB |
| 8 | 124 ms | 160 MiB | 1.09 GiB |
| 32 | 484 ms | 171 MiB | 5.12 GiB |
Latency and memory scale linearly with concurrency. This is the Discord tier-up announce pattern: everyone arrives at once, and the cluster does N identical aggregations.
Two independent reasons nothing is cached today:
ChReadClient.querycallschReadSettings(SCORE_READ_CEILING_S)with nocacheTtlS. Only/metricsopts into ClickHouse's server-side query cache; every score read skips it.initialFillmintsasOfasparseAsOf(url) ?? new Date()— a fresh millisecond per request. Even with the cache enabled, the key would never repeat. This is precisely the unquantized-bound trap already documented inread-protocol.ts.
Proposal
Three pieces:
- Quantize
asOfto a bucket instead ofnew Date(), so the cache key repeats across viewers. - Add a monotonic
{epoch:UInt64}bound param to the board reads, carried as an inert predicate. This is the bust lever. - Pass
cacheTtlSthroughchReadSettingson the board read path.
The freshness contract
Caching is only acceptable if significant events bust it. Verified on ClickHouse 24.10 — a bound param does change the query-cache key:
| sequence | result |
|---|---|
epoch=1 |
miss, 3.73M rows |
epoch=1 |
hit, 5 rows |
epoch=2 |
miss, 3.73M rows |
epoch=2 |
hit, 5 rows |
epoch=1 again |
hit (old entry still live) |
Note the last row: epochs must be monotonic. Reusing an old value re-hits its stale entry, so a counter or timestamp, never a toggle.
Wiring: TierCrossingWatchDeps.onTierReached already exists as an injected hook, and the tier watcher already performs an atomic Mongo compare-and-advance in claimCrossing. Bumping an epoch doc in that same write is nearly free. The board API reads it behind a ~1s per-pod cache — the same shape as the existing enterpriseHiddenUsernames cache in repos.ts, so no new pattern.
Result: cached hard between crossings, provably fresh at them. Eager-tick-plus-announcer-poke, not a lazy TTL.
Expected effect
N concurrent viewers collapse to 1 query per replica (3 across the cluster). Measured on the 10B synthetic with a snapshot table behind it: 64-way concurrency served in 27 ms total wall, 0.6 ms average.
Even without phase 3's snapshot, this alone removes the concurrency multiplier at current scale.
Constraints and traps
- The
asOfecho-back contract must survive quantization. The fill returns the pin and every later page fetch echoes it rather than re-pinning, or rows and the pagination that follows them come from different instants (#1055). - The query cache is per-node on a 3-replica cluster, so expect 3 misses per epoch, not 1. That is fine and worth stating so it isn't read as a bug.
- Confirm
query_cache_min_query_runsandquery_cache_min_query_durationdefaults on the prod nodes are permissive enough to admit these queries, and that the cache size ceiling accommodates the entry. - Errors and ceiling hits are not cached by ClickHouse, so a degraded read cannot get stuck in the cache.
Verification
- A call-counting assertion in the spirit of
tests/routes/leaderboard_initial_fill_test.ts— N page loads must produce 1 upstream read, not N. - Drive a real tier-up through the local stack and assert the board reflects the new standing within one tick, not one TTL.
- Assert epoch monotonicity explicitly; a non-monotonic epoch is a silent staleness bug.
Risk / rollback
Low. Rollback is deleting the cacheTtlS argument — the epoch param is inert on its own.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.