Skip to main content
← Back to list
01Issue
FeatureOpenSwamp Club
AssigneesNone

Relationships

#875 Score-refresh consumer: harden coalescing against a diverse backlog (bistability)

Opened by keeb · 6/28/2026

Problem

The score-refresh consumer (lib/app/process-score-batch.ts + lib/infrastructure/event-queue-watcher.ts, draining the swamp_club_events queue) was hardened in #772 with per-user coalescing, an atomic claim token, and a larger claim window (batchSize 200). That cleared a ~64k-entry backlog at ~25x the prior drain rate and is working well in production.

But the coalescing throughput is bistable, and the current fast drain only holds because the backlog happens to be dominated by a single firehose user.

Each tick: the watcher claims the oldest 200 pending entries, coalesceByUsername collapses them to one refresh per distinct username, and each distinct user is then a sequential block of ~15 MongoDB (Atlas) round-trips (the per-user loop is not parallelized).

  • Firehose-dominated backlog (today): the oldest-200 window is ~94% one user, coalesces to a handful of refreshes, cheap tick, drains ~850/min.
  • Diverse backlog (the latent risk): if the oldest-200 window holds ~200 distinct users, coalescing collapses nothing, so the tick must do ~200 x 15 ~= 3000 sequential round-trips (multi-minute) while draining only 200 raw entries. The producer can then outpace the drain, the window stays diverse, and it becomes self-sustaining.

This is the same metastable collapse that turned a transient trigger into a ~22h queue wedge (drain frontier froze at 2026-06-27T08:56:49Z when a manual ClickHouse v1->v2 cutover briefly slowed the score path; the pre-#772 consumer could not recover). #772 removed the firehose-dominated failure mode but leaves the diverse-backlog mode latent.

Proposed solution

Make per-tick throughput independent of the distinct-user count in the window:

  1. Bounded-concurrency fan-out: process the per-user refreshes within a tick with a concurrency limit (e.g. 8-16 in flight) instead of strictly sequentially. Refreshes are independent except for OCC on each user's own user_scores doc, so bounded parallelism is safe and cuts wall-time ~10x in the diverse case.
  2. Decouple claim width from refresh count: keep claiming a wide window (so firehose duplication still coalesces) but cap distinct-users-refreshed-per-tick, leaving the rest for subsequent ticks.
  3. Observability: add an SLI + alert on queue depth and oldest-pending age so a forming wedge is visible long before it is 64k deep.

Alternatives considered

  • Larger batchSize alone: does not help the diverse case (still N distinct x sequential round-trips).
  • Per-user sharded queues: heavier; unnecessary if fan-out is bounded.

Context

Follow-up to #771 (presence-grant high-water guard) and #772 (coalesce + atomic claim + batch 200). #772 fixed recovery from a deep backlog; this issue hardens the consumer so a diverse backlog cannot recreate the wedge.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

6/28/2026, 4:21:24 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.