Skip to main content
← Back to list
01Issue
FeatureOpenSwamp ClubPublic
AssigneesNone

Relationships

#4050 Streak projector: incremental runs instead of a full scan every minute

Opened by keeb · 10/27/2026

Follow-up to swamp-club PR #1432, which memoizes the streak board, profile score-cards and roster streaks into Mongo (streak_standings, streak_projection).

Problem

Every minute the projector rescans every owner's full day history in ClickHouse (ClickHouseLeaderboardQuery.streakStandings()). Most of that work recomputes values that cannot have changed:

  • An owner's streak, card run and activeDays only change when they gain a new active day, or at UTC midnight, when unrenewed streaks break.
  • Within a day, only owners who are active today can move. For an owner who is already active today, only totalEvents keeps changing.

Proposal

  1. Full scan once per UTC day. Run the full scan just after 00:00 UTC, which is the only time streaks break. Today, readers fall back to ClickHouse from midnight until the next run. Scheduling that run at the boundary closes that gap.
  2. Keyed runs during the day. Rescan only owners with activity today, using active_days_by_id/cli_daily rows where day = today(), resolved to owners through the existing keyed cand pattern. Merge their rows into streak_standings, then re-rank the board from the stored standings rather than from a full scan.
  3. Wake on activity. Instead of waiting for the 60s tick, telemetry requests a run when an owner records their first active day, the same way requestScoreRollupRun works. A new streak day then shows in seconds.

Acceptance

  • scripts/harness-streak-standings.ts still passes, with parity against ClickHouse for every owner and for the board.
  • A day-boundary case: a streak that breaks at midnight reads 0 with no fallback gap.
  • Each daytime run's ClickHouse read_rows scales with the number of owners active today, not with the size of all history.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/27/2026, 10:01:13 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

system commented 10/27/2026, 10:01:13 PM

Classified automatically when this issue was filed.

  • Source: Swamp Club

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.