#4003 Add Axiom-queryable phase telemetry for worker definition deletion and prune

Problem and scope

Split from #3986, which originally covered five expensive Swamp core paths. Each path now has its own issue:

1. enrollment: Enrollment preparation #3986
2. step_outputs: Run-detail output resolution #4002
3. worker_delete: Worker definition deletion and prune #4003 (this issue)
4. output_retention: Output retention #4004
5. access_poll: Access-data polling #4005

Add exported phase telemetry for worker definition deletion and prune so an Axiom query can explain its cost without source-level profiling or local debug logs.

This is a core observability feature. Reuse the tracing/phase-accounting contract in #3786 and existing spans. This issue specifies the internal breakdown that remains missing; it does not request another queue implementation, dashboard, exporter, or performance fix. Inspect current main before adding spans so recently added coverage is reused.

The shared diagnostic phase vocabulary and helpers (area/phase selectors, outcome and coverage fields) are established by #3986. Build on that contract rather than defining a parallel one; if #3986 has not landed yet, coordinate so both use the same schema.

Evidence motivating the instrumentation

Observed on October 26, 2026, production service swamp-serve-vm, build 20261025.114930.0-sha.b4dd513d, filesystem datastore engine v2. This is a historical elapsed timing on that build, not a claim that the current-main path is still slow.

Representative trace ID: 65985d2e42d59eea4400a9ec7bc15688

Required exported contract

Use normal OpenTelemetry spans exported by Swamp's existing pipeline and queryable in Axiom's swamp-serve-traces dataset. Console logs, an in-memory profiler, local counters, or span events that do not survive the actual exporter/Axiom mapping are insufficient.

• Preserve trace_id, span_id, parent_span_id, _time, duration, service identity, runtime version and existing datastore identity. Correlate new work with the initiating request/transition/batch/poll. If work is shared or detached, expose the owning operation and supported span links/IDs rather than misparenting it.
• Reuse the existing outcome/cancellation and phase-coverage conventions. Use monotonic elapsed timing. Finish spans on errors and cancellation, with bounded outcome/error codes. A failed or cancelled phase must remain queryable.
• Distinguish elapsed time from cumulative work across parallel items. Do not derive exclusive elapsed time by summing overlapping child durations or subtracting percentile aggregates. Preserve explicit uncovered/partial coverage where applicable.
• Proposed common selectors: swamp.diagnostic.area = worker_delete (one of the five areas: enrollment, step_outputs, worker_delete, output_retention, access_poll); swamp.diagnostic.phase = a documented static phase name. Reuse an existing equivalent if available, and update the APL examples to the final schema. These are bounded categories, not dynamically constructed span names.
• Export counters as numeric attributes, flags as booleans, and outcomes/cache states as bounded strings. Keep IDs on trace records where needed, not metric labels. Do not emit tokens, secret values, raw resource contents, query payloads or absolute user paths.
• Always retain the stage summary for a retained trace. Bound per-item detail: aggregate large scans/deletion loops into stage spans with counts and cumulative timings; optional sampled item spans must report sampling/truncation. Do not create one unbounded event or attribute per file.
• Document units, counter meanings (attempted/succeeded/unique), supported datastore branches and absent/unsupported fields. Respect existing telemetry opt-out and sampling settings.

Worker definition deletion and prune

Parents: swamp.worker.prune and its swamp.model.delete children.

Add:

• Prune worker listing and eligibility selection, with considered/eligible/skipped counts and bounded skip reasons.
• Definition lookup; workflow-reference discovery/checks; data-artifact discovery and expiry classification; output discovery.
• Data/output removal, definition/evaluated-record removal, and relevant index/catalog updates as distinct phases.
• Post-delete token/binding cleanup, separately timed from worker deletion.
• Attempted/deleted/failed counts, bytes if known, item type, cancellation/partial outcome and batch size. Preserve sequential versus concurrent execution attribution.

Acceptance question: for a 50-minute pass, can APL identify whether the cost is repeated discovery/reference scans, deletion I/O, index maintenance, or binding cleanup? These are worker record/definition deletions; do not label them cloud VM destruction.

Relevant code: src/libswamp/worker/prune.ts, src/libswamp/models/delete.ts, src/serve/worker_gc_service.ts and the repositories called by these paths.

Axiom query deliverables

Commit runnable APL examples with the final schema, not just a screenshot. These examples show the proposed selectors; field names must be reconciled with the implementation.

Phase latency by phase, outcome and runtime:

['swamp-serve-traces']
| where ['service.name'] == 'swamp-serve-vm'
| extend area=tostring(['attributes.custom']['swamp.diagnostic.area']),
         phase=tostring(['attributes.custom']['swamp.diagnostic.phase']),
         outcome=tostring(['attributes.custom']['swamp.outcome']),
         version=tostring(['resource.custom']['swamp.version']),
         ms=duration/1ms
| where area == 'worker_delete'
| summarize n=count(), p50_ms=percentile(ms,50), p95_ms=percentile(ms,95), max_ms=max(ms)
  by phase, outcome, version
| sort by p95_ms desc

A single operation's exported breakdown:

['swamp-serve-traces']
| where trace_id == '<trace-id-from-the-integration-run>'
| project _time, name, ms=duration/1ms, trace_id, span_id, parent_span_id,
          attrs=['attributes.custom'], status=['status.code']
| sort by _time asc

Also provide a query comparing prune/delete discovery versus removal time and counts.

Use explicit UTC query windows and document whether selection is by span start or completed-within-window; retain failures, cancellations and late/cutoff-crossing work as separate populations. Do not label a maximum as p95 or add nested durations as wall time.

Acceptance and validation

1. The path emits the required stages through the normal OTLP exporter. Parentage/correlation, numeric fields, outcomes and units survive the exporter-to-Axiom mapping.
2. Reproducible integration fixtures exercise eligible and ineligible workers, a worker with data, outputs and workflow references, and a prune that fails or is cancelled part way. Include error and cancellation cases, and shared/overlapping work where supported.
3. Capture representative exported traces, then run the documented APL against an authorized test dataset/stream with the same schema. Supply trace IDs and query output proving stages, counters and correlations are readable. Production rollout is not required for this issue.
4. Tests assert stage coverage and known fixture counts, including skipped/no-op branches. Timing tests use ordering/controlled attribution rather than brittle real-time latency thresholds.
5. Confirm bounded telemetry volume and acceptable measured overhead at small and large catalog sizes; verify opt-out and that attributes contain no secrets or raw resource payloads.
6. Audit and reuse instrumentation already introduced by #3786, #3984 and #3968. Record the actual tested revision and separate missing instrumentation from missing deployment.

No browser, vault-read or approvals/index-load expansion is requested in this issue.