#3986 Add Axiom-queryable phase telemetry for worker enrollment preparation

Problem and scope

This issue originally covered five expensive Swamp core paths and has been narrowed to the first one. Each path now has its own issue:

1. enrollment: Enrollment preparation #3986 (this issue)
2. step_outputs: Run-detail output resolution #4002
3. worker_delete: Worker definition deletion and prune #4003
4. output_retention: Output retention #4004
5. access_poll: Access-data polling #4005

Add exported phase telemetry for enrollment preparation so an Axiom query can explain its cost without source-level profiling or local debug logs.

This is a core observability feature. Reuse the tracing/phase-accounting contract in #3786 and existing spans. This issue specifies the internal breakdown that remains missing; it does not request another queue implementation, dashboard, exporter, or performance fix. Coordinate with #3984. Inspect current main before adding spans so recently added coverage is reused.

This issue is first of the five and establishes the shared diagnostic phase vocabulary and helpers (area/phase selectors, outcome and coverage fields, the APL conventions). The other four build on it.

Evidence motivating the instrumentation

Observed on October 26, 2026, production service swamp-serve-vm, build 20261025.114930.0-sha.b4dd513d, filesystem datastore engine v2. This is a historical elapsed timing on that build, not a claim that the current-main path is still slow.

Representative trace ID: 819019055cafbad55dac373155cbbcaf

For enrollment, matching RPC ID c965e720-8c34-4d68-a4c0-a32b9ad8fd8d links worker trace 965221ef3333c2c9947111c7065fe536: the worker times out and submits cancellation at 30s while the server completes at 270s. Submission alone does not establish server receipt. First-observed worker-name method calls had an 18.764s median versus 96.541ms for later calls; first observed is not proof of a new definition. New-definition scan behavior was subsequently reproduced in #3984.

Required exported contract

Use normal OpenTelemetry spans exported by Swamp's existing pipeline and queryable in Axiom's swamp-serve-traces dataset. Console logs, an in-memory profiler, local counters, or span events that do not survive the actual exporter/Axiom mapping are insufficient.

• Preserve trace_id, span_id, parent_span_id, _time, duration, service identity, runtime version and existing datastore identity. Correlate new work with the initiating request/transition/batch/poll. If work is shared or detached, expose the owning operation and supported span links/IDs rather than misparenting it.
• Reuse the existing outcome/cancellation and phase-coverage conventions. Use monotonic elapsed timing. Finish spans on errors and cancellation, with bounded outcome/error codes. A failed or cancelled phase must remain queryable.
• Distinguish elapsed time from cumulative work across parallel items. Do not derive exclusive elapsed time by summing overlapping child durations or subtracting percentile aggregates. Preserve explicit uncovered/partial coverage where applicable.
• Proposed common selectors: swamp.diagnostic.area = enrollment (one of the five areas: enrollment, step_outputs, worker_delete, output_retention, access_poll); swamp.diagnostic.phase = a documented static phase name. Reuse an existing equivalent if available, and update the APL examples to the final schema. These are bounded categories, not dynamically constructed span names.
• Export counters as numeric attributes, flags as booleans, and outcomes/cache states as bounded strings. Keep IDs on trace records where needed, not metric labels. Do not emit tokens, secret values, raw resource contents, query payloads or absolute user paths.
• Always retain the stage summary for a retained trace. Bound per-item detail: aggregate large scans/deletion loops into stage spans with counts and cumulative timings; optional sampled item spans must report sampling/truncation. Do not create one unbounded event or attribute per file.
• Document units, counter meanings (attempted/succeeded/unique), supported datastore branches and absent/unsupported fields. Respect existing telemetry opt-out and sampling settings.

Enrollment preparation

Parent: the worker swamp.model.method.run inside swamp.serve.transition operation enroll, before swamp.model.method.setup. Keep the already-present gateway queue wait/depth separate.

Add phase spans or equivalent exported phase records for:

• Definition resolution, with lookup method (direct file, hint/index, scan), hit/miss, requested type and primary/auto source.
• Each actual scan, with files/definitions examined, files read, bytes read, parse/read failures, and read/parse work timing.
• Auto-definition lock acquisition separate from lock-held work, including outcome and wait duration.
• Creation re-check, create/save, and reused-versus-created outcome. Distinguish the initial lookup and the lookup under the creation lock so duplicate scans are measurable.
• Remaining preparation that can consume this gap, such as routing/validation/type resolution, when not already attributed by an existing phase.
• Cancellation/deadline observed at the server and by the preparation operation, tied to the existing RPC/attempt identity. Preserve separate client-submit/server-observed milestones; do not claim cancellation was received solely because the client sent it.

Acceptance question: can APL distinguish a slow new-name lookup from lock contention, definition save, a warm lookup, and waiting behind another gateway transition?

Relevant code: src/serve/worker_gateway.ts, src/libswamp/worker/run_deps.ts, src/libswamp/models/run.ts, src/libswamp/models/direct_execution.ts, src/infrastructure/persistence/yaml_definition_repository.ts. Lookup is already type-scoped; do not describe or instrument every worker lookup as an all-types scan.

Axiom query deliverables

Commit runnable APL examples with the final schema, not just a screenshot. These examples show the proposed selectors; field names must be reconciled with the implementation.

Phase latency by phase, outcome and runtime:

['swamp-serve-traces']
| where ['service.name'] == 'swamp-serve-vm'
| extend area=tostring(['attributes.custom']['swamp.diagnostic.area']),
         phase=tostring(['attributes.custom']['swamp.diagnostic.phase']),
         outcome=tostring(['attributes.custom']['swamp.outcome']),
         version=tostring(['resource.custom']['swamp.version']),
         ms=duration/1ms
| where area == 'enrollment'
| summarize n=count(), p50_ms=percentile(ms,50), p95_ms=percentile(ms,95), max_ms=max(ms)
  by phase, outcome, version
| sort by p95_ms desc

A single operation's exported breakdown:

['swamp-serve-traces']
| where trace_id == '<trace-id-from-the-integration-run>'
| project _time, name, ms=duration/1ms, trace_id, span_id, parent_span_id,
          attrs=['attributes.custom'], status=['status.code']
| sort by _time asc

Also provide a query comparing enrollment lookup hit versus miss, created versus reused, and scan cost versus definition count.

Use explicit UTC query windows and document whether selection is by span start or completed-within-window; retain failures, cancellations and late/cutoff-crossing work as separate populations. Do not label a maximum as p95 or add nested durations as wall time.

Acceptance and validation

1. The path emits the required stages through the normal OTLP exporter. Parentage/correlation, numeric fields, outcomes and units survive the exporter-to-Axiom mapping.
2. Reproducible integration fixtures exercise new and existing worker definitions, concurrent enrollments contending for the auto-definition lock, and a worker that cancels before the server finishes. Include error and cancellation cases, and shared/overlapping work where supported.
3. Capture representative exported traces, then run the documented APL against an authorized test dataset/stream with the same schema. Supply trace IDs and query output proving stages, counters and correlations are readable. Production rollout is not required for this issue.
4. Tests assert stage coverage and known fixture counts, including skipped/no-op branches. Timing tests use ordering/controlled attribution rather than brittle real-time latency thresholds.
5. Confirm bounded telemetry volume and acceptable measured overhead at small and large catalog sizes; verify opt-out and that attributes contain no secrets or raw resource payloads.
6. Audit and reuse instrumentation already introduced by #3786, #3984 and #3968. Record the actual tested revision and separate missing instrumentation from missing deployment.

No browser, vault-read or approvals/index-load expansion is requested in this issue.