Relationships
#3929 Showcase reward backfill can double-pay 5,000,000 points when replicas start together
Opened by keeb · 10/25/2026
From the web review on #1412, with the risk corrected upward.
lib/app/showcase/showcase-reward.ts rewardShowcase checks the ledger (findAward) and then writes the grant (writeAward). It isn't atomic. The review called the backfill one-time, but it isn't: getShowcaseRepo() in lib/app/repos.ts launches runShowcaseRewardBackfill on every replica's first request after each start, so during a rollout every replica runs it at the same moment.
The ledger doesn't absorb the race. swamp.score_grants is a ReplacingMergeTree ordered by (distinct_id, granted_at, grant_id), and each write stamps its own granted_at = new Date(). Two concurrent writes of award:showcase:<showcaseId> therefore stay as two rows, and the operative is paid 10,000,000 instead of 5,000,000.
Exposure today is narrow: the backfill only pays a live showcase whose reward never landed, which happens when a reward fails at approval time. Every approval that rewards normally is skipped by findAward. But the window opens on every deploy.
Fix options:
- Run the backfill under
withLease(services/telemetry/lib/leader-lease.ts), as AGENTS.md asks for background ledger writers, so only one replica pays. - Make the grant's
granted_atdeterministic (e.g. the showcase'sreview.reviewedAt) so a duplicate write collapses under ReplacingMergeTree.
Doing both closes the approval-time path as well, where an admin double-clicking Approve on two replicas could race the same way.
Open
No activity in this phase yet.
Sign in to post a ripple.