#4004 Add Axiom-queryable phase telemetry for output retention

Problem and scope

Split from #3986, which originally covered five expensive Swamp core paths. Each path now has its own issue:

1. enrollment: Enrollment preparation #3986
2. step_outputs: Run-detail output resolution #4002
3. worker_delete: Worker definition deletion and prune #4003
4. output_retention: Output retention #4004 (this issue)
5. access_poll: Access-data polling #4005

Add exported phase telemetry for output retention so an Axiom query can explain its cost without source-level profiling or local debug logs.

This is a core observability feature. Reuse the tracing/phase-accounting contract in #3786 and existing spans. This issue specifies the internal breakdown that remains missing; it does not request another queue implementation, dashboard, exporter, or performance fix. Inspect current main before adding spans so recently added coverage is reused.

The shared diagnostic phase vocabulary and helpers (area/phase selectors, outcome and coverage fields) are established by #3986. Build on that contract rather than defining a parallel one; if #3986 has not landed yet, coordinate so both use the same schema.

Evidence motivating the instrumentation

Observed on October 26, 2026, production service swamp-serve-vm, build 20261025.114930.0-sha.b4dd513d, filesystem datastore engine v2. This is a historical elapsed timing on that build, not a claim that the current-main path is still slow.

Representative trace ID: dcccfbc54ccfa6ae86af55930f493e42

Required exported contract

Use normal OpenTelemetry spans exported by Swamp's existing pipeline and queryable in Axiom's swamp-serve-traces dataset. Console logs, an in-memory profiler, local counters, or span events that do not survive the actual exporter/Axiom mapping are insufficient.

• Preserve trace_id, span_id, parent_span_id, _time, duration, service identity, runtime version and existing datastore identity. Correlate new work with the initiating request/transition/batch/poll. If work is shared or detached, expose the owning operation and supported span links/IDs rather than misparenting it.
• Reuse the existing outcome/cancellation and phase-coverage conventions. Use monotonic elapsed timing. Finish spans on errors and cancellation, with bounded outcome/error codes. A failed or cancelled phase must remain queryable.
• Distinguish elapsed time from cumulative work across parallel items. Do not derive exclusive elapsed time by summing overlapping child durations or subtracting percentile aggregates. Preserve explicit uncovered/partial coverage where applicable.
• Proposed common selectors: swamp.diagnostic.area = output_retention (one of the five areas: enrollment, step_outputs, worker_delete, output_retention, access_poll); swamp.diagnostic.phase = a documented static phase name. Reuse an existing equivalent if available, and update the APL examples to the final schema. These are bounded categories, not dynamically constructed span names.
• Export counters as numeric attributes, flags as booleans, and outcomes/cache states as bounded strings. Keep IDs on trace records where needed, not metric labels. Do not emit tokens, secret values, raw resource contents, query payloads or absolute user paths.
• Always retain the stage summary for a retained trace. Bound per-item detail: aggregate large scans/deletion loops into stage spans with counts and cumulative timings; optional sampled item spans must report sampling/truncation. Do not create one unbounded event or attribute per file.
• Document units, counter meanings (attempted/succeeded/unique), supported datastore branches and absent/unsupported fields. Respect existing telemetry opt-out and sampling settings.

Output retention

Parent: swamp.retention.batch, retention.kind=outputs.

Split:

• Candidate enumeration/discovery.
• Eligibility/expiry checks and protection checks.
• Deletion and associated index/catalog maintenance.
• Any meaningful lock/gate wait, separately from active work.

Record considered/eligible/protected/skipped/delete-attempted/deleted/failed counts, deletion bytes where known, scan/index/cache strategy, batch size, and completion/cancellation/error outcome. Preserve the current retention.deleted counter's meaning. Partial work before an error must be represented honestly.

Acceptance question: can APL explain why deleting 868 outputs takes nearly 14 minutes, and distinguish scan-heavy batches from deletion-heavy ones without assuming all parent time is file deletion?

Relevant code: src/libswamp/retention/pass.ts and the output-retention repository/service implementations it invokes.

Axiom query deliverables

Commit runnable APL examples with the final schema, not just a screenshot. These examples show the proposed selectors; field names must be reconciled with the implementation.

Phase latency by phase, outcome and runtime:

['swamp-serve-traces']
| where ['service.name'] == 'swamp-serve-vm'
| extend area=tostring(['attributes.custom']['swamp.diagnostic.area']),
         phase=tostring(['attributes.custom']['swamp.diagnostic.phase']),
         outcome=tostring(['attributes.custom']['swamp.outcome']),
         version=tostring(['resource.custom']['swamp.version']),
         ms=duration/1ms
| where area == 'output_retention'
| summarize n=count(), p50_ms=percentile(ms,50), p95_ms=percentile(ms,95), max_ms=max(ms)
  by phase, outcome, version
| sort by p95_ms desc

A single operation's exported breakdown:

['swamp-serve-traces']
| where trace_id == '<trace-id-from-the-integration-run>'
| project _time, name, ms=duration/1ms, trace_id, span_id, parent_span_id,
          attrs=['attributes.custom'], status=['status.code']
| sort by _time asc

Also provide a query comparing output-retention scan/eligibility versus deletion/index work.

Use explicit UTC query windows and document whether selection is by span start or completed-within-window; retain failures, cancellations and late/cutoff-crossing work as separate populations. Do not label a maximum as p95 or add nested durations as wall time.

Acceptance and validation

1. The path emits the required stages through the normal OTLP exporter. Parentage/correlation, numeric fields, outcomes and units survive the exporter-to-Axiom mapping.
2. Reproducible integration fixtures exercise eligible, protected and unexpired outputs, an empty batch, and a batch that errors after deleting some outputs. Include error and cancellation cases, and shared/overlapping work where supported.
3. Capture representative exported traces, then run the documented APL against an authorized test dataset/stream with the same schema. Supply trace IDs and query output proving stages, counters and correlations are readable. Production rollout is not required for this issue.
4. Tests assert stage coverage and known fixture counts, including skipped/no-op branches. Timing tests use ordering/controlled attribution rather than brittle real-time latency thresholds.
5. Confirm bounded telemetry volume and acceptable measured overhead at small and large catalog sizes; verify opt-out and that attributes contain no secrets or raw resource payloads.
6. Audit and reuse instrumentation already introduced by #3786, #3984 and #3968. Record the actual tested revision and separate missing instrumentation from missing deployment.

No browser, vault-read or approvals/index-load expansion is requested in this issue.