#1249 Telemetry stats consumer: unsanitized subcommand in Mongo update path drops user metrics silently

Summary

The telemetry stats consumer builds MongoDB update paths by interpolating client-supplied invocation.command and invocation.subcommand without escaping. A subcommand containing a . produces a malformed path and the batch write throws. Because the consumer's dedup step commits before the failing write, the retry is a no-op that reports success — so the affected user's metrics are lost silently, with no dead-letter and no alert on the loss itself.

What triggers it

swamp init . (or swamp init <path>).

The CLI records the first non-flag token after the command as subcommand, without validating it against a real subcommand list. But init has no subcommands — it takes an optional path positional. So the literal . is emitted as invocation.subcommand, and the update path renders as:

event_counts.cli_invocation.[HOST-1]..

Two consecutive dots — an empty field name. MongoDB rejects it with error 56, "contains an empty field name, which is not allowed".

init is the only command in the CLI tree with no subcommands but a positional arg, so it is currently the only route to this exact crash.

Observed in production

Dataset swamp_club_prod, 2026-07-18. Two incidents:

• [IP-1].959Z — 45 events, 4 distinct_ids
• [IP-2].262Z — 6 events

exception.slug: consumer-stats-publish-failed. Fired the alerts "[sc] Telemetry consumer dispatch failure" and "[sc] Record-of-record consumer error". First occurrence in 7 days. Different batch composition and 33 minutes apart, so these are two separate invocations, not one message redelivering. The queue was otherwise healthy throughout.

Why it matters: silent, unreported data loss

This is the important part, and it is worse than a stuck batch.

_publish runs its insert_id dedup step before the upsert. That dedup write commits. The upsert then fails and throws, and the batch is marked failed and retried. On retry, the dedup step sees every insert_id as a duplicate, so the fresh set is empty, the function returns early with outcome duplicate, and the consumer reports success. The batch is marked delivered and deleted.

Confirmed in the traces:

[IP-1].959  event.count: 45   status: ERROR
[IP-3].490  event.count: 45   outcome: duplicate

Consequences:

• The write never happened, but the pipeline reports green.
• The batch never reaches the attempts >= 5 dead-letter parking in retryFailed, because attempt 2 "succeeds".
• No double-counting (the $incs do not replay) and no poison-message loop — but also no signal that anything was lost.

Blast radius is scoped to the offending device: the per-user update document is built as a single $inc per distinct_id, so one bad key poisons that user's entire update for the batch, not just the init event. Other distinct_ids in the same batch commit normally under ordered: false.

Raw events are preserved — the ClickHouse consumer dispatched independently and shows no errors, and the S3 archive is intact. Only the aggregate counters in user_metrics are affected, so they are rebuildable.

Why this is not a regression

The defect has been present since the feature was introduced in #34 (2026-02-24). sanitizeKey was applied to the event name from the start; the two interpolations of client input never received it.

Notably, the same bug class was hit the same day and patched narrowly in #38, whose commit message cites the identical error:

Empty strings in invocation.command produce invalid MongoDB dot-notation paths like event_counts.cli_invocation.commands..total which fail with "contains an empty field name".

That fix added !== "" guards on both fields. A subcommand of . passes those guards cleanly and reproduces the same failure. So this is a recurrence of an incompletely-fixed bug — the guard enumerated the one bad input that had bitten rather than escaping the path. Still present on main at d7b5605e.

Second, quieter failure mode

A dotted subcommand that is not exactly . — for example a typo like swamp model foo.bar — yields a valid nested Mongo path rather than an error, writing a nested object where a counter belongs. That corrupts the event_counts shape with no crash and no alert. This follows from dot-notation semantics; I have not confirmed it against production data.

Affected components

• services/telemetry/lib/consumers/stats.ts:365 and :373 — path construction from unsanitized client input
• services/telemetry/lib/consumers/stats.ts:427-434 — rethrow on any non-"Cannot create field" write error, failing the whole batch
• services/telemetry/lib/schema.ts:138-148 — ingest validates that properties is a plain object but passes its contents through unexamined

Suggested fix approach

Server-side, in this repo. Apply the same sanitization already used for the event name to command and subcommand at the point of path construction. That closes both the hard-crash and the silent-nesting variants in one change, and removes the reliance on enumerating bad inputs that caused this to recur.

This must be fixed server-side rather than in the CLI. The CLI is a distributed binary, so a client-side fix does nothing about already-installed copies, which keep sending . until users upgrade. Ingest currently passes properties through unvalidated, and events carry no CLI version field, so the affected client population can neither be scoped nor gated. The server fix takes effect for all clients and all versions on next deploy.

A companion change in the swamp CLI — not reporting a subcommand for commands that have none, so swamp init . records command=init, args=["."] — is worth doing for data quality, but it does not fix this and should not block it.

Related but separate concern

The ordering issue is more general than this bug: the dedup write commits before the work it guards, so any failure between those two points is permanently swallowed and the retry reports success. A transient Mongo error during the upsert would behave identically — one alert on the first failure, then silence, with the data gone. This routes around the dead-letter design in retryFailed, which explicitly reasons about not losing score grants during a consumer outage. Probably deserves its own issue.