#3786 serve/fleet: close OpenTelemetry attribution gaps across dispatch, data plane and worker lifecycle

Objective

Close the remaining OpenTelemetry attribution gaps across Swamp core serve, dispatch, data plane and worker lifecycle with one documented, exported telemetry contract. Make a slow or unfinished attempt explainable from admission through settlement without manual timestamp/name joins. This is a core feature request, not an extension request.

Reasoning

All 19 investigated UAT runs have attributable serve and worker traces, but run-level coverage is not complete phase coverage. Shared release traces, nested workflows, worker logs and persisted results currently require manual joins. Missing instrumentation prevents distinguishing scheduling, transport, preparation, persistence and acknowledgment delay. Observations below motivate measurement; they do not establish a storage, GC, lock, fleet-capacity or runtime-regression root cause.

Measured evidence

Baseline: 2026-10-21 00:00:00 UTC ≤ span start < 23:05:00 UTC, additionally requiring span end ≤ 23:05:00 UTC. Observed production runtime: 20261021.050521.0-sha.bfde0792. Dataset swamp-serve-traces; production serve swamp-serve-vm, outer workers swamp-worker; candidate/test-local swamp-uat is a separate population. These are observed completed-span distributions, explicitly right-censored at cutoff; in-flight or never-exported work is absent. Axiom estimated percentiles use duration / 1s; max is separate. Nested spans and overlapping aggregate shard-minutes are not additive wall time.

• Query RPC: 4,995 selected production data-plane spans across 19 traces comprise fetch_bundle 2,578, write_resource 1,241, open_writer 588 and writer_content 588, with no query operation. The checked production span inventory also lacks a separate query RPC span. Test-local query spans cannot substitute; production query latency is unavailable, not zero.
• Pre-dispatch and enrollment: run 42cae35d-e3a5-411f-b58f-0cddff9155b1, trace ddc7b1997cb15f40d01738b0f301e734: readiness pre-dispatch median 99.534s; six steps spend approximately 99–100s before their 120s remote-dispatch deadlines. All 36 workers eventually enrolled, including six 62–92s after readiness deadlines. Worker logs contain 59 giving-up records, 56 naming a 30,000ms enrollment RPC timeout; these are records, not distinct VMs. Across 36-slot runs, pre-dispatch medians reach 143.743s. Outer and inner enrollment p99 values cannot be subtracted to derive an overhead percentile.
• Post-worker settlement: run c2e766d5-e851-47a7-9cdf-8b803036f042, trace 52ca74223f126944d5ba6bed33ec2e92: post-worker-to-step intervals include serve-4 679.814s, cli-6 677.468s and GCS-2 677.398s. Latest completed run a85f3cb9-d067-4062-95bb-ef449dba5e2f, trace ef4b4ba9d56acd388862e53916ed5204, has 71.43 aggregate shard-minutes before worker start and 3.39 after worker finish. These are overlapping exposure, not calendar savings.
• Commit/ACK ambiguity: in c2e766d5, S3-1 and S3-2 persisted all-pass results (46/0 and 36/0 passed/failed, both ok=true and timedOut=false), but steps fail with Serve did not answer POST /data/resource within 900s; serve may still have saved this write. Result-to-worker-completion intervals are 905.039s and 905.303s. S3-1 worker execution span 1020efa575e15c18, parent 5bb233a4046e0c57, in trace 52ca74223f126944d5ba6bed33ec2e92. No dedicated outer-runner write/ACK span was observed; the attribution comes from errors and timestamps. Two separate body-read failure waves lack 19 result resources despite worker completion events.
• Webhook correlation: historical reception lacked delivery IDs; four queue-full logs cannot establish per-delivery receive-to-run latency. Existing merged queue fields do not by themselves provide that join.
• Identity/export: runtime and fleet resource labels already exist. A separate Oct 22 16:25:14.409–17:25:14.409 UTC identity query found 8,729 serve and 250 worker spans identifying bfde0792; this is emitting-runtime evidence, not a complete fleet census or a latency comparison. No matching Swamp p99 panel appeared in the credential-scoped dashboard listing; platform dashboards remain platform-owned.

Evidence sources in platform-swamp: research_notes/UAT release time loss analysis/{latency-percentiles,control-plane,test-failures,upstream-fix-status}.md and reports/swamp-serve-fleet-improvements.html#missing-otel. The notes retain exact APL, source-result filenames, service filters, timing methodology and run/trace joins. Relevant upstream incident evidence is also in 3755.

Scope — missing instrumentation checklist

• Production query RPC: correlate the outer-worker query client with the production capability/query handler; expose request queue wait, service execution and response serialization/transfer/receipt timing, outcome, returned item count and bytes. Distinguish zero results from failure and partial/cancelled response; preserve request linkage on timeout. Do not record query text or returned payloads.
• Canonical causal identities: document and propagate workflow run, nested-run relationship, job/step, attempt, dispatch, lease, stable worker identity and connection/ownership epoch across admission, dispatch, completion and reconnect. Preserve trace context or explicit span links at asynchronous boundaries; distinguish logical work from retries. Link old/new connection and owner epochs without claiming restart survival is already implemented. Coordinate reconnect behavior with #3679; this checklist concerns its telemetry contract, not a second restart-recovery implementation.
• Disjoint pre-dispatch and post-worker phases: expose step admission, scheduler wait, definition/dependency preparation, evaluation/runtime preparation, worker allocation, lease queue/acquire, send, worker receipt/start/end, result receipt, lease completion and workflow-step settlement. Reuse existing setup/transition spans. Define mutually exclusive accounting intervals and explicitly report uncovered time; nested child durations must not be summed. Distinguish user reporting/finalization from controller settlement and account for cancellation and retries.
• Upload/body, commit and ACK: join outer-worker body receive/read/upload/write and response-body lifecycle with server request receipt, lease marking, persistence/commit, response send and caller receipt. Record byte counts, phase outcome, timeout/cancellation and immutable request/result/commit identities where available. Correlate a server commit with a caller timeout and any later reconciliation; distinguish committed, not committed and unknown. Expose missing/unsupported reconciliation explicitly. This instrumentation must not silently introduce write retries, promise idempotency or reinterpret a persisted successful verdict as assertion failure.
• Enrollment lifecycle and retry phases: link worker-start/connect and attempt/deadline to server receive/auth/queue/transition/response and schedulable state; record retry delay, attempt number, cancellation/abandonment, late-response reconciliation and terminal reason. Preserve stable worker identity across process/connection retries. VM allocation and external boot milestones may be linked if supplied, not fabricated by core. Instrument existing behavior without asserting the old dead-session bug #3405 recurred.
• Webhook delivery-to-run correlation: join receive/enqueue/start/reject outcomes using safe delivery identity, route identity, timestamps and resulting run ID, including replay and concurrent same-route deliveries. Use explicit links across intentionally detached traces rather than reparenting unrelated executions. Add rejection reason and elapsed queue interval beyond the merged queue fields. Preserve existing isolation and deduplication semantics; generate an internal receipt ID when the provider supplies none.
• In-flight/error/timeout telemetry and schema: export in-flight counts, oldest pending age, phase latency histograms and bounded outcome/error/timeout/cancel counters for dispatch, transition, query, writes/ACK, enrollment and webhook admission as appropriate. Include cutoff-crossing attempts and sampling/export loss in coverage accounting. Reuse existing queue state; define units, histogram buckets, attribute names, lifecycle/reset semantics, resource identity and supported export configuration in a versioned/documented contract. Keep existing service/fleet/runtime identity consistent across new signals; define bounded fleet and controlled active-version cardinality policies. High-cardinality run/step/attempt/dispatch/lease/worker/delivery/commit IDs belong only in spans/logs or exemplars, never metric labels. No secrets, tokens, raw payloads or unrestricted error strings in telemetry.

Native OTel waterfall requirement — serve to worker and back

The initial deliverable must be visible directly in a standard OpenTelemetry trace waterfall, without a custom dashboard, timestamp matching, or manual log joins. Some propagation already exists; verify and repair gaps rather than assuming all distributed context is absent.

• Propagate W3C traceparent / tracestate in each dispatch envelope, extract it before worker execution spans start, and propagate the active execution context back on worker-to-serve RPCs. Preserve supported sampling decisions and context through subprocess launch. Validate untrusted context at ingress and preserve per-dispatch isolation.
• For causally nested job execution, use the same trace ID and correct parent span IDs across services. Matching run IDs or span links alone are not sufficient for this waterfall requirement. Keep distinct service/resource and worker-instance identities.
• Represent actual phases, for example (names illustrative; reuse existing conventions):

workflow.run                         [serve]
  step.execute                       [serve]
    dispatch.queue                   [serve]
    dispatch.send                    [serve]
      worker.prepare                 [worker]
      worker.execute                 [worker]
        data.write                   [worker]
          data.persist               [serve]
      completion.accept              [serve]

• For a cold worker provisioned by the workflow, carry causal context through supported provisioning metadata so startup, connect, enrollment request, serve acceptance and schedulable state can be followed from that provisioning operation. Only instrument milestones actually observed; do not fabricate external boot spans.
• A warm/shared worker's earlier startup is not a child of a later workflow. Keep its lifecycle trace separate and add explicit OTel span links to the dispatch that selects it. Jobs on a reused worker must not inherit another job's trace; reconnect and intentionally detached work retain documented links where parentage would be misleading.

Additional acceptance: export one cold-worker run and one warm-worker run to an ordinary OTel-compatible trace viewer. The job's serve → dispatch → worker execution → serve RPC → completion path must render with correct cross-service nesting in one trace, without manual joins. The cold-worker provisioning/lifecycle path must preserve supported causal context; the warm-worker lifecycle must be discoverable through exported span links (viewer support may vary). Automated assertions must verify trace IDs, parent IDs, links, distinct concurrent-job contexts, RPC return-path context, and supported subprocess/sampling propagation. This is native telemetry visibility, not a custom UI or a new worker-recovery guarantee.

Acceptance

1. A controlled 36-worker readiness/query/small-write burst produces a reproducible joined timeline. ≥99% of dispatch attempts with linked, disjoint accounting through settlement or terminal timeout/cancellation is a proposed benchmark gate, not achieved coverage. Publish numerator/denominator, uncovered intervals, still-in-flight count/age and dropped/sampled telemetry explicitly; do not discard unsuccessful attempts to meet the gate.
2. Integration coverage exercises successful, failed, timed-out, cancelled and cutoff-crossing requests; delayed enrollment and retry; delayed body upload/read; server commit followed by lost response; reconnect/ownership changes supported by the runtime; concurrent webhook delivery, rejection and pending replay. Verify cross-attempt isolation and causal joins, including nested workflows. For unsupported reconnect/reconciliation behavior emit an explicit outcome and retain links.
3. For completed phases report fixed-window n/p50/p95/p99/max with documented units and separate outcome populations. Use raw durations/histograms, never merge per-bin percentiles or label maximum as p99. Retain trace exemplars. Demonstrate that pending age and timeouts remain visible while no completed span exists.
4. Provide core schema/export documentation and a reproducible benchmark/query example. Verify bounded metric series and no high-cardinality IDs or sensitive data in metric labels/payloads. Exercise export on/off and record instrumentation overhead under matched load; hold worker count, payload size, datastore/history volume and runtime revision fixed for before/after comparisons. This is telemetry verification, not a new optimization/SLO promise or custom dashboard build.
5. Validate the already-merged instrumentation in a runtime containing the fixes before implementing missing pieces. Record effective versions/configuration separately from merge status. Platform-owned dashboards can consume the contract; creating those dashboards and production rollout are not upstream implementation asks here.

Related / rollout-only dependencies

• Lab 3755, merged PR 3280, Oct 21 21:51:52 UTC: already adds swamp.serve.transition with tail, operation, queue_wait_ms and queue_depth, plus split swamp.serve.data_plane.mark_writes and .persist, and improves lease-read scaling. Deploy/verify and reuse; do not implement these spans again. Local benchmark improvements do not quantify historical production attribution.
• Lab 3665, merged PR 3281, Oct 21 22:02:28 UTC: already adds webhook concurrency and per-route active, queued, oldestQueuedAt, lastQueueDelayMs, plus effective concurrency config (default 1). Rollout/config verification only for those fields. Remaining work here is delivery/receipt-to-run telemetry and rejection linkage, coordinated with 3665's identity requirement.
• 3679: existing restart/reattachment feature owns execution-survival semantics; share its correlation contract.
• 3519/PR 3162: longer write deadline already shipped; measured 900s failures motivate ACK visibility, not another deadline increase.
• 1911, 2589, 2445: shipped query-routing/parenting/test-correlation fixes are related, not evidence of their recurrence.
• 949: existing pending-dispatch introspection should be reused. 1084 and 2239 already provide resource/version identity. 3741 owns shipped history-read tracing; no duplicate history optimization request.