GET /api/v1/users/NAME/score-history?detail=full burns roughly 5 seconds of ClickHouse and returns zero entries, for every operative tested — not just the device-heavy skew identities the existing notes describe. The slim variant of the same window returns full data in under a second. The profile day drill-down (the per-day event mix / combat log detail) is therefore dead UI that silently renders as "no data" rather than as an error.

Found while investigating the slow profile card (see PR #1055, which fixed a different 5s read on the same page).

Evidence (production, 2026-08-11)

Request	Result
keeb, days=90, slim	90 entries in 0.40s
keeb, days=90, detail=full	0 entries in 4.80s
webframp, days=90, slim	data in 0.63s
webframp, days=90, detail=full	0 entries in 5.10s

Same operative, same window, same instant. The only difference is detail=full, which is the arm that pays the raw-events event-mix scan.

Downstream: /u/webframp/activity measures 1.9s to 3.7s across runs (variance consistent with a cache-dependent full scan), versus 1.0s for keeb.

Cause

This is the residual documented in infrastructure/clickhouse/CLAUDE.md under "Quest-read latency: the distinct_id scan trap (Lab #1317)". dailyHistory.eventMix is the last request-time raw-events read on the profile path. swamp.events is ORDER BY (event, created_at) with only a distinct_id bloom index, so a per-identity filter has no primary-key access path and the read scans the partition, breaching the 5s max_execution_time.

The failure is invisible because loadProfileScore catches a ledger read failure and degrades to an empty standing. A timeout therefore renders as "no activity" rather than as an outage — the same "a 5s failure that looks like no data" shape the doc already calls out.

The docs understate the blast radius

infrastructure/clickhouse/CLAUDE.md frames the residual as specific to an operative whose devices touch every granule, naming webframp at 1187 devices, and says pruning fixed the general case (measured 335ms to 14ms). Production disagrees: keeb (182 active days, roughly 570k events, not device-heavy) fails identically at 4.80s with 0 entries. So this is not a skew-identity tail — the detail=full read appears to be failing broadly. That claim in the doc should be corrected as part of this fix.

Suggested fix

The doc already prescribes it: pruning cannot fix this shape, only a rollup can. Build a (distinct_id, day, event, command, subcommand)-keyed materialized view — the quest_signal_counts treatment plus a day dimension — and switch the read to it.

Ordering matters and is non-negotiable: the CREATE TABLE, CREATE MV and one-shot idempotent backfill must reach production as a clickhouse-migration (hash bumped, applied) BEFORE the code that reads the new object deploys. Otherwise the read 404s UNKNOWN_TABLE on every call, which is the 2026-07-06 ordered-change trap that froze the activity projector for 16 hours.

Two smaller things worth doing in the same change:

Make the drill-down distinguish "still loading / degraded" from "no activity in this window", so a ceiling breach reports an outage instead of quietly claiming the operative did nothing. Same trap as the leaderboard fill, where pending had to be checked before the empty-rows branch.
Consider whether detail=full should be served at all while it cannot answer within the ceiling — right now every call costs roughly 5 seconds of a 2-vCPU node and returns nothing, which is pure load with no product value.

02Bog Flow

Shipped

8/11/2026, 1:38:04 AM

Click a lifecycle step above to view its details.

03Sludge Pulse

keeb assigned keeb8/11/2026, 12:19:31 AM

Only documented non-interactive auth puts the password in argv; the safe SWAMP_API_KEY path is undocumented

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

Docs: update cancel endpoint response statuses in REST API reference and workflows page

docs: update vault audit trail reference for write operation support

docs: document --dashboard flag for swamp serve

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

Relative path for custom agent skills setup doesn't provision skills

swamp serve daemon enable: add --dashboard flag

CLI writes runtime state (lastStalenessWarning) into tracked .swamp.yaml, dirtying a committed config file

extension install serializes files[] in readdir order, producing machine-dependent reorder diffs in upstream_extensions.json

Workflow YAML rewritten in place: block-scalar description truncated, file left unparseable, and validate still reports PASSED

workflow validate does not resolve modelType on factory-pattern steps — a non-existent extension validates clean

vault.get() arguments are verbatim, so a per-target secret cannot be expressed in workflow YAML

Cancellation/--timeout skips remaining steps including `completed`/`always` branches; --timeout waits for the in-flight subprocess

Remote "write-bearing" classification is inferred from data-plane writes, not declared; external side effects can be re-executed on worker disconnect (docs say the opposite)

Audit timeline is not populated by swamp CLI commands unless the AI-tool hook fires (manual says recording is automatic)

manual_approval suspends the run while parallel sibling jobs are mid-flight; in-flight steps are torn down and left `running`

`workflow resume --from` does nothing for forEach steps whose template name contains `${{ }}`

Step-level dependsOn on a forEach step never evaluates true (except `always`)

rum/site-info: lookup GETs a non-existent endpoint, always 405

vault read-secret appends a newline to piped output

wss:// direct to serve's own TLS listener still fails on 20260817 — client ALPN lands on h2, WS-over-h2 unimplemented (follow-up to #1363)

serve: vault audit repository not wired up — audit trail always empty

Support Jira webhooks with X-Hub-Signature HMAC-SHA256 verification

Add typed read-only remote ref lookup to @swamp/git

pullChanged skips same-size updated files — size-only change detection ignores the index sha256

A run's terminal status reaches the shared datastore only when a later run happens to push it

workflow cancel --all on the serve host shuts down the serve process

Readability of tables in documentation

repo init/upgrade --tool kiro references .kiro/skills/ but installs skills to ~/.kiro/skills/

vault: audit trail should record put, delete, and annotate operations

data.findBySpec() returns only the first in-run record, not all of them, so an .all() assert silently checks one element

repo upgrade --tool kiro reports wrong changedFiles paths

Suppress boot-time "pulled extension(s) have missing source files" WRN when invoked command is `swamp extension install`

s3-datastore: a stale machine's sync pushes its old 'latest' pointer over the newer remote one, and the rollback is permanent

`search` → `list` aliases are hand-rolled Command clones, not aliases: 10 of 11 have drifted from their parent's option set

Make official Docker image OCI-compliant

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

data prune still misclassifies live records as orphans on 20260820 (after #1396) — serve-minted token/grant records, and report data of every live workflow

vault put prompts for the secret before validating the vault name exists

workflow validate: false 'Missing required inputs' for CEL-templated globalArgs (${{ inputs.* }})

Add SWAMP_CLI_OUTPUT env var to control default output mode (plaintext/tui/json)

@swamp/git fetch: support local path as remote

Dispatched steps over 15 minutes cannot write results: dispatch credentials expire and never refresh

Concurrent pushes to one namespace fail with HTTP 409 ConditionalRequestConflict — the conditional PUT is never retried

serve: triggerSource not persisted to workflow run summary

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

--max-run-duration only applies to detached runs, not attached runs

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve

Docs: update workflow reference to include trigger in JSON output shapes