Skip to main content
← Back to list
01Issue
BugShippedSwamp Club
Assigneeskeeb

Relationships

#1249 Telemetry stats consumer: unsanitized subcommand in Mongo update path drops user metrics silently

Opened by swamp_lord · 7/18/2026· Shipped 7/18/2026

Summary

The telemetry stats consumer builds MongoDB update paths by interpolating client-supplied invocation.command and invocation.subcommand without escaping. A subcommand containing a . produces a malformed path and the batch write throws. Because the consumer's dedup step commits before the failing write, the retry is a no-op that reports success — so the affected user's metrics are lost silently, with no dead-letter and no alert on the loss itself.

What triggers it

swamp init . (or swamp init <path>).

The CLI records the first non-flag token after the command as subcommand, without validating it against a real subcommand list. But init has no subcommands — it takes an optional path positional. So the literal . is emitted as invocation.subcommand, and the update path renders as:

event_counts.cli_invocation.[HOST-1]..

Two consecutive dots — an empty field name. MongoDB rejects it with error 56, "contains an empty field name, which is not allowed".

init is the only command in the CLI tree with no subcommands but a positional arg, so it is currently the only route to this exact crash.

Observed in production

Dataset swamp_club_prod, 2026-07-18. Two incidents:

  • [IP-1].959Z — 45 events, 4 distinct_ids
  • [IP-2].262Z — 6 events

exception.slug: consumer-stats-publish-failed. Fired the alerts "[sc] Telemetry consumer dispatch failure" and "[sc] Record-of-record consumer error". First occurrence in 7 days. Different batch composition and 33 minutes apart, so these are two separate invocations, not one message redelivering. The queue was otherwise healthy throughout.

Why it matters: silent, unreported data loss

This is the important part, and it is worse than a stuck batch.

_publish runs its insert_id dedup step before the upsert. That dedup write commits. The upsert then fails and throws, and the batch is marked failed and retried. On retry, the dedup step sees every insert_id as a duplicate, so the fresh set is empty, the function returns early with outcome duplicate, and the consumer reports success. The batch is marked delivered and deleted.

Confirmed in the traces:

[IP-1].959  event.count: 45   status: ERROR
[IP-3].490  event.count: 45   outcome: duplicate

Consequences:

  • The write never happened, but the pipeline reports green.
  • The batch never reaches the attempts >= 5 dead-letter parking in retryFailed, because attempt 2 "succeeds".
  • No double-counting (the $incs do not replay) and no poison-message loop — but also no signal that anything was lost.

Blast radius is scoped to the offending device: the per-user update document is built as a single $inc per distinct_id, so one bad key poisons that user's entire update for the batch, not just the init event. Other distinct_ids in the same batch commit normally under ordered: false.

Raw events are preserved — the ClickHouse consumer dispatched independently and shows no errors, and the S3 archive is intact. Only the aggregate counters in user_metrics are affected, so they are rebuildable.

Why this is not a regression

The defect has been present since the feature was introduced in #34 (2026-02-24). sanitizeKey was applied to the event name from the start; the two interpolations of client input never received it.

Notably, the same bug class was hit the same day and patched narrowly in #38, whose commit message cites the identical error:

Empty strings in invocation.command produce invalid MongoDB dot-notation paths like event_counts.cli_invocation.commands..total which fail with "contains an empty field name".

That fix added !== "" guards on both fields. A subcommand of . passes those guards cleanly and reproduces the same failure. So this is a recurrence of an incompletely-fixed bug — the guard enumerated the one bad input that had bitten rather than escaping the path. Still present on main at d7b5605e.

Second, quieter failure mode

A dotted subcommand that is not exactly . — for example a typo like swamp model foo.bar — yields a valid nested Mongo path rather than an error, writing a nested object where a counter belongs. That corrupts the event_counts shape with no crash and no alert. This follows from dot-notation semantics; I have not confirmed it against production data.

Affected components

  • services/telemetry/lib/consumers/stats.ts:365 and :373 — path construction from unsanitized client input
  • services/telemetry/lib/consumers/stats.ts:427-434 — rethrow on any non-"Cannot create field" write error, failing the whole batch
  • services/telemetry/lib/schema.ts:138-148 — ingest validates that properties is a plain object but passes its contents through unexamined

Suggested fix approach

Server-side, in this repo. Apply the same sanitization already used for the event name to command and subcommand at the point of path construction. That closes both the hard-crash and the silent-nesting variants in one change, and removes the reliance on enumerating bad inputs that caused this to recur.

This must be fixed server-side rather than in the CLI. The CLI is a distributed binary, so a client-side fix does nothing about already-installed copies, which keep sending . until users upgrade. Ingest currently passes properties through unvalidated, and events carry no CLI version field, so the affected client population can neither be scoped nor gated. The server fix takes effect for all clients and all versions on next deploy.

A companion change in the swamp CLI — not reporting a subcommand for commands that have none, so swamp init . records command=init, args=["."] — is worth doing for data quality, but it does not fix this and should not block it.

The ordering issue is more general than this bug: the dedup write commits before the work it guards, so any failure between those two points is permanently swallowed and the retry reports success. A transient Mongo error during the upsert would behave identically — one alert on the first failure, then silence, with the data gone. This routes around the dead-letter design in retryFailed, which explicitly reasons about not losing score grants during a consumer outage. Probably deserves its own issue.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

7/18/2026, 4:09:48 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb7/18/2026, 3:17:04 PM
Editable. Press Enter to edit.

swamp_lord commented 7/18/2026, 10:06:19 AM

Alerting temporarily tuned down — revert when this is fixed

Two Axiom monitors were adjusted on 2026-07-18 to stop this bug paging while the fix is pending. Both should be reverted as part of the fix PR.

1. [sc] Record-of-record consumer error (6dHzymIhTM7eSSkFJp)

Type unchanged (MatchEvent, threshold 0, interval 1m, range 1m). Added an exclusion clause to the APL:

and not (tostring(["status.message"]) contains "contains an empty field name")

Validated before applying: across 7 days this suppresses exactly the 2 events from this bug and preserves 3 unrelated consumer errors. Stats publish failures from any other cause still alert normally.

Revert: drop the and not (...) clause, restoring the query to the four consumer.*.publish names with ["error"]==true.

2. [sc] Telemetry consumer dispatch failure (xrpHuMQzIWg4ShVTrE)

Converted from MatchEvent to Threshold, because the [HOST-1] span carries no exception slug or message — it only records consumer.failures, so there is nothing on it that distinguishes this bug from any other consumer failure.

before after
type MatchEvent Threshold
operator — Above
threshold 0 2
rangeMinutes 1 10
intervalMinutes 1 1
aplQuery filter only filter + summarize count() (columnName: count_)

Notifier blZ92RIxUwHpZRIoSC preserved on both.

Revert: restore MatchEvent with threshold 0 / range 1m, and drop the trailing | summarize count() and columnName.

Coverage gaps this opens in the meantime

  • Any future "empty field name" Mongo error in the four record-of-record consumers is now silent — including the command-side variant of this same bug. This issue is the only tracker for it.
  • One or two dispatch failures inside a 10-minute window no longer alert. The 7-day baseline is 5 total, so this is low-volume by design — but the 4 consumers not covered by the record-of-record monitor have lost their single-failure signal.

Neither gap is dangerous while this issue is open, but the exclusion will otherwise quietly outlive the bug it was written for. That is precisely the failure mode that let #38's narrow guard survive five months into this incident, so it is worth undoing deliberately rather than leaving in place.

keeb commented 7/18/2026, 4:10:02 PM

Thanks @swamp_lord for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.