Lab #731: Telemetry not synced to swamp-club: local queue accumulating ~3 days despite valid auth

Description

The swamp CLI records command telemetry locally but has stopped uploading/syncing it to swamp-club. The local telemetry queue under <repo>/.swamp/telemetry/ has been accumulating for roughly 3 days with nothing being drained, indicating the upload path has silently stalled. Local recording itself is healthy — only the remote sync is affected.

Affected component

The telemetry upload/sync path that drains .swamp/telemetry/*.json to swamp-club. (Local capture/recording is working.)

Evidence / diagnosis

Local recording works: swamp doctor audit -> overallStatus "pass", including the recording-smoke-test ("synthetic claude payload landed in today's audit JSONL").
swamp telemetry stats shows 188 invocations recorded at 100% success — data is being captured.
Local queue is growing, not draining: <repo>/.swamp/telemetry/ holds ~198 telemetry-*.json files spanning 2026-06-19 -> 2026-06-21; the oldest unsynced file is dated 2026-06-19. None appear to be pruned or marked synced after upload.
Not an auth problem: swamp auth whoami reports an authenticated session against https://swamp-club.com.
No telemetry sync-state / cursor file exists under .swamp/ (only .datastore-sync-state.json for datastore sync), so the drain failure leaves no visible error or progress marker.
Individual telemetry files contain no synced / uploadedAt field, so sync progress is not observable locally.

Environment

swamp 20260620.200156.0-sha.d9abe1a2 (up to date; update --check -> up_to_date)
deno 2.8.3
platform: linux
autoupdate disabled; non-interactive agent session

Expected fix approach (high level)

The telemetry uploader appears to fail silently and stop draining the local queue. The fix would involve: surfacing upload failures (log + retry with backoff) instead of silently halting; persisting a sync cursor / last-synced timestamp so backlog size and last error are observable (e.g. via swamp telemetry stats or swamp doctor); pruning files only after confirmed upload; and adding a swamp doctor check for telemetry upload health (last successful sync time, pending backlog count) so this is detectable going forward.

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

swamp repo upgrade does not remove the superseded skills it warns about

Extension safety scanner rejects .eval() method calls, blocking @swamp/stagecraft install

swamp serve does not register webhook schemes from pulled extensions; boot fails with misleading 'already installed but failed to load'

Subscription-end email wrongly says 'trial ended' after paid subscription

Hot-reload serve.yaml webhook routes (like trigger overrides), not just at startup

Add a static-token webhook scheme (constant-time header equality) for Telegram-style secret tokens

writeCatalogExport takes 55-150s per workflow step — serializes entire catalog as pretty-printed JSON

buildDeps should reuse repoContext.unifiedDataRepo instead of creating a standalone repo per step

readResource returns stale cursor on serve scheduled workflow runs

serve boot: sweepStaleRecords does a full modelMethodRun per stale lease (~12.5s each), serializing boot reconciliation — 4m28s to sweep nothing

install.sh: 'stable' alias tarball and .sha256 are published non-atomically — checksum verification fails during every release window

swamp extension quality resolves datastores: against the repo root but models: against the manifest dir — datastore extensions are unscorable in a monorepo

`--server` list commands render the TUI picker into a non-TTY pipe

`search` → `list` aliases are hand-rolled Command clones, not aliases: 10 of 11 have drifted from their parent's option set

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

Feed: detect and gracefully handle dead (404) post links (link rot)

/feed: per-post unread indicators and mark-as-read to track reading progress

Website extension search ignores API relevance — exact name match "good-planning" ranks #10 (API ranks it #1)

Lab: URLs inside list items aren't clickable (linkified) in rendered issue bodies/summaries

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

extension quality --json exits 1 with 'rubric factors missing' even when a full score was computed (unearned factor, not uncomputable)

seadex-latent-bugs: latent product bugs pinned by the wave-2b test backfill

data delete removes data locally but push reports 'no changes' — next datastore sync restores everything

No way to select which vault receives a sensitive-marked field's value

workflow assert: a step whose CEL expression fails to evaluate is omitted from --junit output, so the XML reports a clean pass

worker daemon enable: relative/missing --cache-dir fails launchd EX_CONFIG while daemon status reports running

swamp doctor extensions: nondeterministic BundleBuildFailed for local extensions on an unchanged tree

Issue redactor corrupts reports: dotted code identifiers masked as hostnames, loopback and RFC 5737 documentation IPs masked, angle-bracket placeholders half-eaten

data prune resolves auto-definitions at the local path, not the datastore path — deletes live models' data (verified: destroyed a working server token and an active grant)

workflow/job-lifetime data is never reclaimed — expiry check requires an ownerDefinition.workflowId that step execution never sets

runModel breadth limit (MAX_INVOCATION_BREADTH=100) is never enforced — tracking object is never written back to the caller context; verified 150/150 calls succeeded

serve daemon enable is ungated — it reports success, then the account gate fires inside the detached daemon, which crash-loops invisibly

Purely local operations require a swamp-club account: local-token serve is gated while deprecated --auth-mode none is not, third-party OIDC is gated, and 'Local_Encryption' is paywalled by capitalization

A collective token stored in auth.json (rather than SWAMP_API_KEY) is permanently scope-less — resolution only runs for the env var, and scopes are never persisted for collective tokens

An empty stored apiKey counts as authenticated — saveIdentityCache writes apiKey:'' on a virgin machine, permanently satisfying the mandatory-account gate with no credential

Account/scope gates fire before argument, type and repo validation — 'swamp vault create env my-vault' (the command's own example, an invalid type) returns a paywall instead of 'Unknown vault type'

auth token create advertises a 'collective:write' scope that does not exist, and following its advice hits 'collective tokens cannot create other collective tokens'

requireScope does exact string matching — fine-grained scopes fail gates that demand the literal wildcard, and minted scopes are never validated or normalized

Collective-token scope cache never expires, is never refreshed by whoami, and cannot be cleared via the CLI — scope changes and revocations never take effect

Collective-token scope cache is keyed on a 12-char token prefix — only 2 chars distinguish org tokens, so a rotated token silently inherits another token's scopes

Misleading 'Your collective token lacks the X scope' when scope resolution failed — startup whoami errors are silently swallowed

Community points / reputation: award points for upvoting extensions, feature requests, comments, and feed posts

datastore setup extension drops 'namespace' from --config, making extensions that require it impossible to configure

Remote --server wss:// fails through an HTTP/2 TLS reverse proxy (WS client ALPN is h2-only)

Published extension scores dependency-trust as "missing" (0/2) while client `extension quality` earns it (2/2); no verdict exposed

namespace unset --migrate silently deletes artifact history via catalog-rebuild GC (~484k files lost)

Bugs filed by a team-subscription user are not tagged "team" in the Lab

command/shell renders JSON-object stdout lines as the literal string 'undefined'

Issue redactor treats the MongoDB authSource parameter as a secret, corrupting reproduction steps

Collective invite is never claimed when the invitee registers via Google with the invited address — stays pending forever, invisible to the invitee

/orgs shows no pending collective invitations to the invitee — no list, banner, or accept affordance anywhere in the product

Cannot subscribe: personal collective has 0 operatives (owner missing from own roster) — payment returns Forbidden, while whoami and extension push both grant the collective

quest: 'Case Closed' (issue_shipped) does not fire when an issue you filed ships — #1181 shipped, challenge still 0/1 (bug_hunter fired for the same issue)

Unreachable path in .swamp-sources.yaml is fatal to every command, including 'swamp version'

OTel setup manual example (OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317) cannot work — exporter is OTLP/HTTP-only, needs port 4318

Configure the OTLP trace exporter (endpoint/service/headers) in .swamp.yaml, not only via OTEL_ env vars

swamp serve leaks esbuild bundler subprocesses as zombies (reparented to PID 1, never reaped)

`swamp extension search --channel beta|rc|stable` ignores the filter — all channels + the no-flag default return an identical result set

Beta-channel extensions are never scored — perpetual 'Scoring is taking longer than expected'

swamp serve scheduled execution runs steps in an isolated data scope: readResource sees empty state, writeResource never reaches the model-scope chain — stateful scheduled workflows duplicate instead of deduplicating

swamp serve: model resource vault-backed sensitive fields resolve empty during served/workflow execution (readResource)

swamp serve --auth-mode token: redeem reads token-main as null → every valid token rejected 'Authentication failed' (real error hidden at debug)

swamp serve: hardcoded WebSocket host allowlist makes it unusable behind a reverse proxy (no --trusted-hosts/--allowed-origins)

swamp issue bug times out posting to the Lab while swamp-club.com returns HTTP 200

telemetry stats fatally fails to load an installed datastore extension (auto-resolve path); all other commands load it fine

Trajectory chart: current-day x-axis label is clipped at the right edge

Telemetry not synced to swamp-club: local queue accumulating ~3 days despite valid auth

Eager render stack (Ink/yoga-layout + marked-terminal) adds ~540ms to every command's startup

Telemetry flush blocks command exit, adding ~1.2s p50 to every invocation

model type describe --json: bloated output (40% duplicated specs, no compact mode) and lost method-to-output mapping drive agents to read extension source

Yanked extension version still shown as active on swamp-club.com and in 'extension search'