#4005 Add Axiom-queryable phase telemetry for access-data polling

Problem and scope

Split from #3986, which originally covered five expensive Swamp core paths. Each path now has its own issue:

1. enrollment: Enrollment preparation #3986
2. step_outputs: Run-detail output resolution #4002
3. worker_delete: Worker definition deletion and prune #4003
4. output_retention: Output retention #4004
5. access_poll: Access-data polling #4005 (this issue)

Add exported phase telemetry for access-data polling so an Axiom query can explain its cost without source-level profiling or local debug logs.

This is a core observability feature. Reuse the tracing/phase-accounting contract in #3786 and existing spans. This issue specifies the internal breakdown that remains missing; it does not request another queue implementation, dashboard, exporter, or performance fix. Inspect current main before adding spans so recently added coverage is reused.

The shared diagnostic phase vocabulary and helpers (area/phase selectors, outcome and coverage fields) are established by #3986. Build on that contract rather than defining a parallel one; if #3986 has not landed yet, coordinate so both use the same schema.

Evidence motivating the instrumentation

Observed on October 26, 2026, production service swamp-serve-vm, build 20261025.114930.0-sha.b4dd513d, filesystem datastore engine v2. This is a historical elapsed timing on that build, not a claim that the current-main path is still slow.

Representative trace ID: 8d85c79dc2ae7556e4f71b22bec8fd64

Required exported contract

Use normal OpenTelemetry spans exported by Swamp's existing pipeline and queryable in Axiom's swamp-serve-traces dataset. Console logs, an in-memory profiler, local counters, or span events that do not survive the actual exporter/Axiom mapping are insufficient.

• Preserve trace_id, span_id, parent_span_id, _time, duration, service identity, runtime version and existing datastore identity. Correlate new work with the initiating request/transition/batch/poll. If work is shared or detached, expose the owning operation and supported span links/IDs rather than misparenting it.
• Reuse the existing outcome/cancellation and phase-coverage conventions. Use monotonic elapsed timing. Finish spans on errors and cancellation, with bounded outcome/error codes. A failed or cancelled phase must remain queryable.
• Distinguish elapsed time from cumulative work across parallel items. Do not derive exclusive elapsed time by summing overlapping child durations or subtracting percentile aggregates. Preserve explicit uncovered/partial coverage where applicable.
• Proposed common selectors: swamp.diagnostic.area = access_poll (one of the five areas: enrollment, step_outputs, worker_delete, output_retention, access_poll); swamp.diagnostic.phase = a documented static phase name. Reuse an existing equivalent if available, and update the APL examples to the final schema. These are bounded categories, not dynamically constructed span names.
• Export counters as numeric attributes, flags as booleans, and outcomes/cache states as bounded strings. Keep IDs on trace records where needed, not metric labels. Do not emit tokens, secret values, raw resource contents, query payloads or absolute user paths.
• Always retain the stage summary for a retained trace. Bound per-item detail: aggregate large scans/deletion loops into stage spans with counts and cumulative timings; optional sampled item spans must report sampling/truncation. Do not create one unbounded event or attribute per file.
• Document units, counter meanings (attempted/succeeded/unique), supported datastore branches and absent/unsupported fields. Respect existing telemetry opt-out and sampling settings.

Access-data polling

Parent: swamp.serve.poll, swamp.serve.poller=access_data.

Add phases for:

• Local change detection, including files/records examined and changes found.
• Synchronization-gate wait and remote pull when that branch is used.
• Catalog invalidation and policy snapshot reload/build.
• Error/retry handling where it consumes meaningful time.

Record source mode (local change source versus sync pull), reload reason, changed/loaded/unreadable record counts, applied-versus-no-change/failed/cancelled outcome, and cache/snapshot reuse when applicable. A local filesystem poll should not fabricate a remote-pull phase.

Acceptance question: can APL distinguish a two-minute file scan from gate contention, a slow remote pull, or rebuilding the authorization snapshot?

Relevant code: src/serve/access_data_poller.ts, the change source, sync-gate helpers and policy snapshot loader.

Axiom query deliverables

Commit runnable APL examples with the final schema, not just a screenshot. These examples show the proposed selectors; field names must be reconciled with the implementation.

Phase latency by phase, outcome and runtime:

['swamp-serve-traces']
| where ['service.name'] == 'swamp-serve-vm'
| extend area=tostring(['attributes.custom']['swamp.diagnostic.area']),
         phase=tostring(['attributes.custom']['swamp.diagnostic.phase']),
         outcome=tostring(['attributes.custom']['swamp.outcome']),
         version=tostring(['resource.custom']['swamp.version']),
         ms=duration/1ms
| where area == 'access_poll'
| summarize n=count(), p50_ms=percentile(ms,50), p95_ms=percentile(ms,95), max_ms=max(ms)
  by phase, outcome, version
| sort by p95_ms desc

A single operation's exported breakdown:

['swamp-serve-traces']
| where trace_id == '<trace-id-from-the-integration-run>'
| project _time, name, ms=duration/1ms, trace_id, span_id, parent_span_id,
          attrs=['attributes.custom'], status=['status.code']
| sort by _time asc

Also provide a query comparing access poll detection versus gate/pull/reload, separated by source mode.

Use explicit UTC query windows and document whether selection is by span start or completed-within-window; retain failures, cancellations and late/cutoff-crossing work as separate populations. Do not label a maximum as p95 or add nested durations as wall time.

Acceptance and validation

1. The path emits the required stages through the normal OTLP exporter. Parentage/correlation, numeric fields, outcomes and units survive the exporter-to-Axiom mapping.
2. Reproducible integration fixtures exercise changed and unchanged polls from a local change source and from a sync pull, unreadable records, and a poll cancelled at shutdown. Include error and cancellation cases, and shared/overlapping work where supported.
3. Capture representative exported traces, then run the documented APL against an authorized test dataset/stream with the same schema. Supply trace IDs and query output proving stages, counters and correlations are readable. Production rollout is not required for this issue.
4. Tests assert stage coverage and known fixture counts, including skipped/no-op branches. Timing tests use ordering/controlled attribution rather than brittle real-time latency thresholds.
5. Confirm bounded telemetry volume and acceptable measured overhead at small and large catalog sizes; verify opt-out and that attributes contain no secrets or raw resource payloads.
6. Audit and reuse instrumentation already introduced by #3786, #3984 and #3968. Record the actual tested revision and separate missing instrumentation from missing deployment.

No browser, vault-read or approvals/index-load expansion is requested in this issue.