#4002 Add Axiom-queryable phase telemetry for workflow run-detail output resolution

Problem and scope

Split from #3986, which originally covered five expensive Swamp core paths. Each path now has its own issue:

1. enrollment: Enrollment preparation #3986
2. step_outputs: Run-detail output resolution #4002 (this issue)
3. worker_delete: Worker definition deletion and prune #4003
4. output_retention: Output retention #4004
5. access_poll: Access-data polling #4005

Add exported phase telemetry for run-detail output resolution so an Axiom query can explain its cost without source-level profiling or local debug logs.

This is a core observability feature. Reuse the tracing/phase-accounting contract in #3786 and existing spans. This issue specifies the internal breakdown that remains missing; it does not request another queue implementation, dashboard, exporter, or performance fix. Reuse the shipped #3968 implementation and its shared-work behavior. Inspect current main before adding spans so recently added coverage is reused.

The shared diagnostic phase vocabulary and helpers (area/phase selectors, outcome and coverage fields) are established by #3986. Build on that contract rather than defining a parallel one; if #3986 has not landed yet, coordinate so both use the same schema.

Evidence motivating the instrumentation

Observed on October 26, 2026, production service swamp-serve-vm, build 20261025.114930.0-sha.b4dd513d, filesystem datastore engine v2. This is a historical elapsed timing on that build, not a claim that the current-main path is still slow.

Representative trace ID: 917dcd9b39a2924ed6498f4ec1a132ce

Required exported contract

Use normal OpenTelemetry spans exported by Swamp's existing pipeline and queryable in Axiom's swamp-serve-traces dataset. Console logs, an in-memory profiler, local counters, or span events that do not survive the actual exporter/Axiom mapping are insufficient.

• Preserve trace_id, span_id, parent_span_id, _time, duration, service identity, runtime version and existing datastore identity. Correlate new work with the initiating request/transition/batch/poll. If work is shared or detached, expose the owning operation and supported span links/IDs rather than misparenting it.
• Reuse the existing outcome/cancellation and phase-coverage conventions. Use monotonic elapsed timing. Finish spans on errors and cancellation, with bounded outcome/error codes. A failed or cancelled phase must remain queryable.
• Distinguish elapsed time from cumulative work across parallel items. Do not derive exclusive elapsed time by summing overlapping child durations or subtracting percentile aggregates. Preserve explicit uncovered/partial coverage where applicable.
• Proposed common selectors: swamp.diagnostic.area = step_outputs (one of the five areas: enrollment, step_outputs, worker_delete, output_retention, access_poll); swamp.diagnostic.phase = a documented static phase name. Reuse an existing equivalent if available, and update the APL examples to the final schema. These are bounded categories, not dynamically constructed span names.
• Export counters as numeric attributes, flags as booleans, and outcomes/cache states as bounded strings. Keep IDs on trace records where needed, not metric labels. Do not emit tokens, secret values, raw resource contents, query payloads or absolute user paths.
• Always retain the stage summary for a retained trace. Bound per-item detail: aggregate large scans/deletion loops into stage spans with counts and cumulative timings; optional sampled item spans must report sampling/truncation. Do not create one unbounded event or attribute per file.
• Document units, counter meanings (attempted/succeeded/unique), supported datastore branches and absent/unsupported fields. Respect existing telemetry opt-out and sampling settings.

Run-detail output resolution

Parent: swamp.workflow.step_outputs.resolve.

Measure:

• Data catalog/repository initialization.
• Output-owner/definition lookup or index construction, with definitions visited, cache hit/miss/rebuild, and whether the caller joined shared work.
• Per-output read authorization, separately from owner resolution; record allowed/denied/error counts. This is distinct from the outer workflow authorization span.
• Content read/materialization and JSON parsing separately, with attempted/successful/missing/error counts, bytes and cumulative timings.
• Nested/child-run lookup and traversal, with calls, unique runs and reused lookups where meaningful.
• Assembly/merge work if material, plus completed/partial/cancelled outcome and resources processed before cancellation.

Keep existing step_outputs.resources_read and resources_denied compatible; document that an attempted read is not necessarily a unique resource or successful read. Ensure one shared ownership scan is not double-counted as both owner resolution and authorization CPU.

Acceptance question: can APL tell whether a slow history request is dominated by ownership lookup, permission decisions, large/slow data reads, parsing, or nested runs?

Relevant code: src/libswamp/workflows/history_get.ts, src/libswamp/workflows/run.ts, src/domain/workflows/step_output_resolver.ts, src/serve/handlers/resource_resolution.ts. Reuse the current #3968 implementation and shared-work behavior.

Axiom query deliverables

Commit runnable APL examples with the final schema, not just a screenshot. These examples show the proposed selectors; field names must be reconciled with the implementation.

Phase latency by phase, outcome and runtime:

['swamp-serve-traces']
| where ['service.name'] == 'swamp-serve-vm'
| extend area=tostring(['attributes.custom']['swamp.diagnostic.area']),
         phase=tostring(['attributes.custom']['swamp.diagnostic.phase']),
         outcome=tostring(['attributes.custom']['swamp.outcome']),
         version=tostring(['resource.custom']['swamp.version']),
         ms=duration/1ms
| where area == 'step_outputs'
| summarize n=count(), p50_ms=percentile(ms,50), p95_ms=percentile(ms,95), max_ms=max(ms)
  by phase, outcome, version
| sort by p95_ms desc

A single operation's exported breakdown:

['swamp-serve-traces']
| where trace_id == '<trace-id-from-the-integration-run>'
| project _time, name, ms=duration/1ms, trace_id, span_id, parent_span_id,
          attrs=['attributes.custom'], status=['status.code']
| sort by _time asc

Also provide a query comparing output resolution ownership/cache costs versus resource counts and bytes.

Use explicit UTC query windows and document whether selection is by span start or completed-within-window; retain failures, cancellations and late/cutoff-crossing work as separate populations. Do not label a maximum as p95 or add nested durations as wall time.

Acceptance and validation

1. The path emits the required stages through the normal OTLP exporter. Parentage/correlation, numeric fields, outcomes and units survive the exporter-to-Axiom mapping.
2. Reproducible integration fixtures exercise authenticated output resolution with allowed and denied outputs, missing and unparseable content, nested runs, and two requests joining one ownership build. Include error and cancellation cases, and shared/overlapping work where supported.
3. Capture representative exported traces, then run the documented APL against an authorized test dataset/stream with the same schema. Supply trace IDs and query output proving stages, counters and correlations are readable. Production rollout is not required for this issue.
4. Tests assert stage coverage and known fixture counts, including skipped/no-op branches. Timing tests use ordering/controlled attribution rather than brittle real-time latency thresholds.
5. Confirm bounded telemetry volume and acceptable measured overhead at small and large catalog sizes; verify opt-out and that attributes contain no secrets or raw resource payloads.
6. Audit and reuse instrumentation already introduced by #3786, #3984 and #3968. Record the actual tested revision and separate missing instrumentation from missing deployment.

No browser, vault-read or approvals/index-load expansion is requested in this issue.