#1346 swamp-club: score-reads tierTotals GET overflows URL with inline param_ownerIds (Invalid URL, mislabeled clickhouse-network-error) for large collectives

Summary

The score-reads tierTotals ClickHouse read builds a GET request with the entire ownerIds array inlined as query params (param_ownerIds=[<UUID>, <UUID>, ...]). For a large collective the URL overflows and fetch/undici throws Invalid URL, before the request ever leaves the pod. This is not a ClickHouse infra problem — CH never sees the request.

Observed (prod, 2026-07-22)

• Onset ~15:00Z, still active ~17:15Z. Tripped [sc] Novel error-slug spike (catch-all) at 17:00Z (41 > 40).
• Two slugs move in perfect lockstep, ~155/day each vs ~1-9/day baseline all prior week (~30x jump):
  • http-client-clickhouse-network-error (scope http-client.clickhouse, span [HOST-2])
  • app-score-reads-fallback-tierTotals-failed (scope [HOST-3])
• 100% of the network-errors have status.message: Invalid URL http://[HOST-4].[HOST-1]:8123/?database=swamp&param_usernames=[evrardjp]&param_ownerIds=[<hundreds of UUIDs>...]
• All 154 failures were the same query — param_usernames=[evrardjp] (one large collective), retried ~150x over 2h, each re-failing.

Impact

Low blast radius but persistent: the affected collective (evrardjp) tierTotals cannot be computed and the fallback also fails, so that profile shows degraded/stale tier data. Not site-wide.

Root cause

ownerIds is passed inline in the GET query string. Above some length the URL is rejected (Invalid URL) and the CH http client mislabels the thrown error as http-client-clickhouse-network-error (same mislabel family as the tier-watch client abort, #1107).

Suggested fix

Send param_ownerIds in the POST body (ClickHouse HTTP supports params in the body) instead of the GET query string, and/or chunk/batch the ownerIds. Consider fixing the mislabel so genuine network errors are distinguishable from client-side URL failures.

Monitor gaps this exposed

1. [sc] ClickHouse timeout spike matches only slug http-client-clickhouse-timeout — blind to http-client-clickhouse-network-error; only the catch-all caught this.
2. Catch-all threshold (40/15min) sits at this condition's noise floor, causing flappy fire/resolve.

Source tag: axiom-triage