← Back to list
01Issue
FeatureOpenSwamp ClubPublic
AssigneesNone
Relationships
#4050 Streak projector: incremental runs instead of a full scan every minute
Opened by keeb · 10/27/2026
Follow-up to swamp-club PR #1432, which memoizes the streak board, profile score-cards and roster streaks into Mongo (streak_standings, streak_projection).
Problem
Every minute the projector rescans every owner's full day history in ClickHouse (ClickHouseLeaderboardQuery.streakStandings()). Most of that work recomputes values that cannot have changed:
- An owner's
streak, cardrunandactiveDaysonly change when they gain a new active day, or at UTC midnight, when unrenewed streaks break. - Within a day, only owners who are active today can move. For an owner who is already active today, only
totalEventskeeps changing.
Proposal
- Full scan once per UTC day. Run the full scan just after 00:00 UTC, which is the only time streaks break. Today, readers fall back to ClickHouse from midnight until the next run. Scheduling that run at the boundary closes that gap.
- Keyed runs during the day. Rescan only owners with activity today, using
active_days_by_id/cli_dailyrows whereday = today(), resolved to owners through the existing keyedcandpattern. Merge their rows intostreak_standings, then re-rank the board from the stored standings rather than from a full scan. - Wake on activity. Instead of waiting for the 60s tick, telemetry requests a run when an owner records their first active day, the same way
requestScoreRollupRunworks. A new streak day then shows in seconds.
Acceptance
scripts/harness-streak-standings.tsstill passes, with parity against ClickHouse for every owner and for the board.- A day-boundary case: a streak that breaks at midnight reads 0 with no fallback gap.
- Each daytime run's ClickHouse
read_rowsscales with the number of owners active today, not with the size of all history.
02Bog Flow
Open
No activity in this phase yet.
03Sludge Pulse
Editable. Press Enter to edit.
system commented 10/27/2026, 10:01:13 PM
Classified automatically when this issue was filed.
- Source: Swamp Club
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.