#2817 CLI telemetry redaction: user values still transmitted (errorMessage, workflowContext, init path, history args, private types) while command words are over-redacted

Summary

CLI telemetry redaction is inconsistent in both directions. It was audited end-to-end against the current CLI, with every payload captured by a loopback sink (method below). The two problems:

• Not comprehensive enough. Ten user-controlled surfaces still send customer-chosen names, paths or queries verbatim. These reach the swamp-club telemetry pipeline, even though our stated position is that usage data carries no customer confidential information.
• Too strict. System-defined values that carry no customer information and that product analytics needs are stripped. 57 of the 192 leaf commands lose their third command word, and built-in type names are redacted in some commands.

The root cause is that redaction is a denylist bolted onto a positional parser. Anything the parser treats as "system-defined" is sent verbatim, whether or not it is. Anything it doesn't recognise is redacted, whether or not it is safe. Free-text fields (errorMessage, workflowContext) sit outside the args logic entirely.

This umbrella issue ties together the earlier point fixes (#1824, #1249, #1235) and the related design and doc issues, and proposes one model to replace them.

Standard being applied

• must-redact: any value a user chose. This covers model, workflow, job, step, vault and repo names, filesystem paths, queries, method inputs, secrets, private extension collectives and types, and free-text error content that interpolates those.
• must-preserve: values that come from swamp itself. This covers CLI command words at any depth, flag names, built-in and public-registry (@swamp/*) type names, error class and code, versions and platform.

How this was measured

A canary-based audit (a swamp extension model, @keeb/redactor, method audit). For each surface it:

1. Builds a throwaway repo with its own HOME and SWAMP_HOME, and a cleared environment.
2. Points .swamp.yaml telemetryEndpoint (and SWAMP_TELEMETRY_ENDPOINT) at a 127.0.0.1 sink.
3. Plants a unique random token (rdx-<case>-<random>) where the user value goes.
4. Runs the real CLI and searches every captured /ingest payload for the token.

It also runs cleanly inside unshare -rn, where only loopback exists. There are 26 cases and 9 opt-out control checks. The case corpus pins current behaviour, so a fix flips the relevant cases and they must be updated in the same change.

Build audited: 20260206.200442.0 (current main, src/cli/telemetry_integration.ts, src/domain/telemetry/error_message_redaction.ts).

Findings: transmitted verbatim (must-redact)

Why these happen:

• 1–4: #1824 added redactErrorMessage(), which only rewrites the username segment after /home/[REDACTED] /Users/[REDACTED] and `C:\Users[REDACTED] and hostnames under a fixed TLD list. It leaves the rest of the path and every interpolated name untouched. Only the first line is kept, but that first line is exactly where the name lives.
• 5: This comes from the #301 design: per-step child events carry workflow, job and step names by design, and only redactErrorMessage is applied to WorkflowTelemetryBridge.finalize(), whose error text also names the model.
• 6: The first non-flag token after the command is always recorded as subcommand, and init takes a path there. #1249 fixed the server-side Mongo crash this caused, but the CLI still transmits the path. That issue's own prod evidence shows real user paths arriving. . is the most common value, but full paths arrive too.
• 7 and 8: ARG_SCHEMAS["model method"] = ["categorical", "redact", "categorical"] was written for run <model> <method>. It applies unchanged to history get|search|logs, where the third positional is a model name, output id or query.
• 9 and 10: Type and method slots are treated as system-defined unconditionally. For anything outside the built-in and public registry, they name the customer's collective and internal tooling.

Verified not transmitted:

• positional names and queries
• flag values (both --k v and --k=v)
• vault put keys and values
• stdout and stderr
• environment variable values (only an allowlisted set of agent-detection booleans is derived)
• the absolute repo path
• the home username
• internal hostnames under the fixed TLD list

Findings: stripped although safe to keep (must-preserve)

Running extractCommandInfo over every leaf of swamp help --json shows 57 of 192 leaf commands lose a categorical command word. This is the gap #1235 describes from the scoring side: workflow run and workflow run search are indistinguishable, as are model method run, describe and history, and aliases key as typed.

Related: opt-out scope

This is not redaction, but it affects what leaves the machine. Verified with the same harness:

There is no single file setting that disables telemetry everywhere. A --repo-dir that doesn't resolve falls back to the default endpoint, and so does any command run from outside the repo. The fallback is to the public default even when the repo's .swamp.yaml sets telemetryEndpoint or telemetryDisabled. There is also no telemetry page in the manual that says any of this (#1597).

There is also no dedicated "build but don't send" mode. It is achievable today by pointing telemetryEndpoint at a closed loopback port, which spools every payload locally, but that isn't documented as a feature.

Proposed direction

Invert the model: allowlist what swamp knows, redact everything else.

1. Command words from the real command tree. Record the resolved Cliffy command path, canonicalised for aliases, as the command identity at any depth; this subsumes #1235. Positionals are never promoted to subcommand, which fixes 6 and the #1249 class.
2. Per-command positional schemas keyed on the full path. Use model method run, not model method, so history get|search|logs default to redact (fixes 7 and 8). Unknown means redact.
3. Type and method names: send as written only when they resolve to a built-in or a public-registry type. Otherwise replace with a stable marker such as <EXTENSION>, or a salted per-install hash if distinct counts matter (fixes 9 and 10).
4. errorMessage: stop sending free text. Send errorType plus a stable error code, or template id, emitted where UserError is constructed. Until codes exist, keep only the message template, with interpolations stripped (fixes 1–4).
5. workflowContext: drop workflowName, jobName and stepName, or replace them with salted hashes so per-run step grouping still works. Keep modelType under rule 3, and runId. Apply the same rule to the bridge's error string (fixes 5).
6. Opt-out scope: make telemetryDisabled also honoured from <config>/telemetry.yaml inside repos, so a user-level opt-out wins everywhere. Consider DO_NOT_TRACK. Document a supported local-only mode, e.g. telemetryEndpoint: none, which spools locally and never sends.
7. Enterprise extensibility: #1674 proposes user-configurable redaction rules for issue submission. The same rule file could feed telemetry, so orgs can add their own codenames and domains.

Acceptance criteria

• A property or conformance test in swamp runs every leaf of the command tree with canary positionals and flag values. It asserts that no canary appears anywhere in the resulting TelemetryEntryData, and that every command word appears in the recorded command identity.
• Cases 1–10 above flip to redacted, and the two over-redaction cases flip to preserved, in the @keeb/redactor audit.
• The error-message and workflow-context changes are covered by tests over the bridge and createErrorResult.
• A manual telemetry page documents the payload fields, redaction rules, every opt-out and its scope, and the local-only mode (#1597).

Related issues

• #1824 (security, shipped): errorMessage sent verbatim. The fix added home-username and internal-host rewriting only. Findings 1–4 are the remainder.
• #1249 (bug, shipped): swamp init <path> puts the path in subcommand. The server was hardened, but the CLI still transmits it (finding 6).
• #1235 (feature, open): fold third-level verbs into subcommand and canonicalise aliases. This is the over-redaction half of this issue.
• #301 (feature, shipped): per-method workflow child events. It introduced workflowContext names (finding 5).
• #1674 (feature, open): user-configurable redaction rules for issue submission. The proposed rule format could extend to telemetry.
• #1597 (feature, open): the manual never mentions telemetry, its fields or its opt-outs.