The score-reads tierTotals ClickHouse read builds a GET request with the entire ownerIds array inlined as query params (param_ownerIds=[<UUID>, <UUID>, ...]). For a large collective the URL overflows and fetch/undici throws Invalid URL, before the request ever leaves the pod. This is not a ClickHouse infra problem — CH never sees the request.

Observed (prod, 2026-07-22)

Onset ~15:00Z, still active ~17:15Z. Tripped [sc] Novel error-slug spike (catch-all) at 17:00Z (41 > 40).
Two slugs move in perfect lockstep, ~155/day each vs ~1-9/day baseline all prior week (~30x jump):
- http-client-clickhouse-network-error (scope http-client.clickhouse, span [HOST-2])
- app-score-reads-fallback-tierTotals-failed (scope [HOST-3])
100% of the network-errors have status.message: Invalid URL http://[HOST-4].[HOST-1]:8123/?database=swamp&param_usernames=[evrardjp]&param_ownerIds=[<hundreds of UUIDs>...]
All 154 failures were the same query — param_usernames=[evrardjp] (one large collective), retried ~150x over 2h, each re-failing.

Impact

Low blast radius but persistent: the affected collective (evrardjp) tierTotals cannot be computed and the fallback also fails, so that profile shows degraded/stale tier data. Not site-wide.

Root cause

ownerIds is passed inline in the GET query string. Above some length the URL is rejected (Invalid URL) and the CH http client mislabels the thrown error as http-client-clickhouse-network-error (same mislabel family as the tier-watch client abort, #1107).

Suggested fix

Send param_ownerIds in the POST body (ClickHouse HTTP supports params in the body) instead of the GET query string, and/or chunk/batch the ownerIds. Consider fixing the mislabel so genuine network errors are distinguishable from client-side URL failures.

Monitor gaps this exposed

[sc] ClickHouse timeout spike matches only slug http-client-clickhouse-timeout — blind to http-client-clickhouse-network-error; only the catch-all caught this.
Catch-all threshold (40/15min) sits at this condition's noise floor, causing flappy fire/resolve.

Source tag: axiom-triage

02Bog Flow

In Progress

7/22/2026, 6:01:39 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

swamp_lord assigned swamp_lord7/22/2026, 5:22:35 PM

Website extension search ignores API relevance — exact name match "good-planning" ranks #10 (API ranks it #1)

Extension/type search at method granularity — find capabilities, not just packages

data query and data get return the same payload under different keys

forEach leaves self.* unresolved in target

It is possible for a member to be a member of a collective twice

swamp-club: score-reads tierTotals GET overflows URL with inline param_ownerIds (Invalid URL, mislabeled clickhouse-network-error) for large collectives

apps sync/lookup fail with 'partial() cannot be used on object schemas containing refinements'

Docker-related model types/workflows would benefit from first-class dry-run/plan support

open method fails for password-auth hosts: SSHPASS env set but ssh call never wrapped in sshpass

Hide enterprise (SSO) operatives from announcements and leaderboard by default

Worker queries need a safe connected-worker filter for fleet fan-out

Published extension scores dependency-trust as "missing" (0/2) while client `extension quality` earns it (2/2); no verdict exposed

Add a :micdrop: emoji / reaction shortcode (swamp-native art included)

Docs: update autoGc documentation to reflect write-time version pruning

namespace unset --migrate silently deletes artifact history via catalog-rebuild GC (~484k files lost)

serve: workflow.approvals and workflow.run.search still full-scan the workflow-run store after the #1284 data.query pushdown

Bugs filed by a team-subscription user are not tagged "team" in the Lab

Bundled extension model cannot deserialize AWS SDK S3 XML that works in direct Deno

gcp/sheets spreadsheets-values: batch_get crashes with "Request with GET/HEAD method cannot have body"

Token creation API and UI scope selector for collective tokens (phase 4 of #960)

Make Cloudflare CRUD methods factory-aware so list-discovered resources can be managed in-place

[sc] quest-grant Mongo ops (recentHeartbeat/signalCounts/countIssueShipped) regressed to ~5s timeouts — missing index

Add @swamp/kubernetes/statefulset model type

Sync guard skips bound namespace directory during push — 0 files pushed after namespace migrate

Flaky #1052 preparePush test: mtime-granularity race on Linux tmpfs

command/shell renders JSON-object stdout lines as the literal string 'undefined'

Issue redactor treats the MongoDB authSource parameter as a secret, corrupting reproduction steps

Docs: document repo root resolution from subdirectories

Expose the swamp.club feed as an RSS feed

Email invitation is not linked to account created from invite URL

Add serve, datastore, vault, and worker wildcard scopes to token registry (phase 4 prep, #960)

datastore setup extension drops 'namespace' from --config, making extensions that require it impossible to configure

Docs: update manual to document --yes/-y as the standard confirmation-skip flag

Docs: update data.query and CEL expressions references for SQL pushdown

Docs: document collective token CLI usage in API key authentication pages

Add a --release-notes option to `swamp update` (or even make it the default)

Docs: manual should state workflow unknown keys are rejected and placement is step-only (follow-up to #1240)

Make GCP CRUD methods factory-aware so list-discovered resources can be managed in-place

serve: read ops (data.query, workflow.approvals, workflow run.search) linearly scan the whole datastore — ~350–600ms floor regardless of match count

Running "swamp workflow run ..." in a subdir of the swamp repo fails

Guide agents to inspect reports after failed model and workflow runs

serve: main JS thread holds ~40–70% of a core while idle (no clients, no runs, --no-schedule)

Search box collapses to near-zero width on mobile on the Extensions page

Extensions page: Search bar missing on mobile

spreadsheets-values: every method fails with 'Missing required path parameter: spreadsheetId' — implementation reads a global argument the schema does not declare

namespace migrate/set workflow broken for extension datastores: migrate skipped by sync guard, set not idempotent, migrate-index empties shards

ExtensionInfo.latestVersion typed 'string' but registry returns null; harden pull/resolve for prerelease-only extensions

workflow.run: relay a custom model method's output as method_output (command steps already do)

promptChoice infinite-loops on non-interactive stdin

gcp _lib: unconditional x-goog-user-project header causes USER_PROJECT_DENIED (403) for service accounts without the serviceUsageConsumer role

extension push: issue-verdict review is indistinguishable from missing/stale

swamp extension version crashes with 'Invalid CalVer version: "null"' when only a prerelease (beta/rc) version is published

Pushing an extension after creating a public git repo that is the same as other previously pushed extensions doesn't update the "Verified public repository"

`datastore setup filesystem` never transfers content when leaving a sync-based (remote) datastore — copies only the catalog and hardcodes `filesPulled: 0`

Scaleway models: create methods should not require the resource ID they generate (ipId/gatewayId/serverId/...)

gcp/calendar events.list reads wrong array field ('defaultReminders' not 'items') → always returns 0 events

provisioner.provision fails with 'Cannot read properties of undefined (reading trim)' at method_execution_service.ts:852

method_execution_service.ts:852 crashes on undefined.trim() when writeResource receives undefined instance name

CLI support for collective API tokens (phase 3 of #960)

@swamp/gcp/calendar: service-account & ADC auth mint cloud-platform-scoped tokens → 403 on Calendar API

Scope enforcement and whoami introspection for collective tokens (phase 2 of #960)

fix(software-factory): DeclaredSchemaSchema.enum rejects boolean literals

Surface cloud-provider labels (scaleway, digitalocean, oracle/oci) as first-class search/filter facets on the Extensions page

Profile link in genesis subdirectory does not navigate back to main profile

Profile stats: per-user daily rollup table + MV in ClickHouse

Reconcile collective-token CLI activity to members (attribute → bonus → stamp → even-split)

Telemetry stats consumer: unsanitized subcommand in Mongo update path drops user metrics silently

Telemetry stats consumer: insert_id dedup commits before the write it guards — any upsert failure is silently swallowed on retry

Job-level labels: block is silently ignored, not rejected

Issue submission redactor silently corrupts report bodies

CLI telemetry: fold third-level verbs into subcommand + canonicalize aliases

Docs: update extension info reference to document extensions (type grafts) in content metadata

Docs: swamp skill repo-structure reference still documents repo-local .swamp/telemetry/ as live

Add safe adoption and lifecycle methods for existing Hetzner resources

Docs: update manual telemetry reference for user-global spool and telemetry.yaml opt-out

export const extension can graft methods onto foreign-collective types, undisclosed in extension info / contentMetadata

Add lookup support to the Cloudflare DNS records model

Expand token scope registry to full taxonomy (phase 1 addendum, #960)

Docs: document user-global telemetry (telemetry.yaml opt-out, spool location, telemetry stats behavior)

score_daily rollup strands stale grains when a day-less grant re-stamps onto a later day

Grant revoke is never applied to the running server — under `--grant-reload auto` and `manual` alike, a revoked grant stays in force until an explicit `access reload`