Skip to main content
← Back to list
01Issue
BugOpenSwamp ClubPublic
AssigneesNone

Relationships

#3929 Showcase reward backfill can double-pay 5,000,000 points when replicas start together

Opened by keeb · 10/25/2026

From the web review on #1412, with the risk corrected upward.

lib/app/showcase/showcase-reward.ts rewardShowcase checks the ledger (findAward) and then writes the grant (writeAward). It isn't atomic. The review called the backfill one-time, but it isn't: getShowcaseRepo() in lib/app/repos.ts launches runShowcaseRewardBackfill on every replica's first request after each start, so during a rollout every replica runs it at the same moment.

The ledger doesn't absorb the race. swamp.score_grants is a ReplacingMergeTree ordered by (distinct_id, granted_at, grant_id), and each write stamps its own granted_at = new Date(). Two concurrent writes of award:showcase:<showcaseId> therefore stay as two rows, and the operative is paid 10,000,000 instead of 5,000,000.

Exposure today is narrow: the backfill only pays a live showcase whose reward never landed, which happens when a reward fails at approval time. Every approval that rewards normally is skipped by findAward. But the window opens on every deploy.

Fix options:

  • Run the backfill under withLease (services/telemetry/lib/leader-lease.ts), as AGENTS.md asks for background ledger writers, so only one replica pays.
  • Make the grant's granted_at deterministic (e.g. the showcase's review.reviewedAt) so a duplicate write collapses under ReplacingMergeTree.

Doing both closes the approval-time path as well, where an admin double-clicking Approve on two replicas could race the same way.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/25/2026, 6:44:03 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.