OPENTELEMETRY

Swamp uses OpenTelemetry (OTel) for distributed tracing and structured log export. Both signals are opt-in and controlled entirely through standard OTel environment variables. When no provider is registered, all tracer, span, and logger operations are no-ops with zero overhead.

Configuration

Telemetry is enabled when OTEL_EXPORTER_OTLP_ENDPOINT is set or either OTEL_TRACES_EXPORTER or OTEL_LOGS_EXPORTER is console. When neither condition is met, no SDK packages are loaded and no async hooks are installed.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 swamp workflow run my-workflow

Setting OTEL_TRACES_EXPORTER=console prints spans to stderr without requiring a collector endpoint. Setting OTEL_LOGS_EXPORTER=console prints log records to stderr in the same way.

Logs

Structured log lines emitted during a run are exported as native OTLP log records, sent to the collector's /v1/logs endpoint. Logs share the same OTEL_EXPORTER_OTLP_ENDPOINT as traces — when the endpoint is set, both signals are active by default.

Trace Correlation

Each log record carries the trace_id and span_id of the active span at the time the log was emitted. This links log lines to the span that produced them, so a collector or backend can display logs inline with their parent trace.

Redaction

Run secrets (vault values, environment variables marked sensitive) are redacted from log record bodies and attributes before export. Redaction happens in the export pipeline, so secrets never leave the process in OTLP payloads.

Coverage

Log export covers swamp serve and worker subprocesses. Worker subprocesses receive trace context via the propagated TRACEPARENT environment variable, so their log records are correlated with the parent trace. That TRACEPARENT comes only from the dispatch's own trace headers, never from the worker's environment — see Traces in swamp serve and workers.

Batching

By default, log records are exported individually (one OTLP request per record). For long-running processes like swamp serve, set OTEL_BLRP_USE=1 to enable the BatchLogRecordProcessor, which buffers records and flushes them periodically — reducing request volume to the collector.

Span Naming Convention

All spans use the prefix swamp. followed by the domain and operation:

swamp.<domain>.<operation>

Span Hierarchy

A workflow run produces a trace with the following structure:

swamp.cli
  ├─ swamp.lock.acquire
  ├─ swamp.datastore.sync
  └─ swamp.workflow.run.command
       └─ swamp.workflow.run
            ├─ swamp.workflow.evaluate
            ├─ swamp.workflow.job
            │    ├─ swamp.workflow.step
            │    │    └─ swamp.model.method
            │    └─ swamp.workflow.step
            │         └─ swamp.model.method
            └─ swamp.workflow.job
                 └─ swamp.workflow.step
                      └─ swamp.model.method

A scheduled/cron run creates a separate root span:

swamp.scheduled.fire
  └─ swamp.workflow.run
       └─ ...

CLI Root Span

Every CLI invocation creates a swamp.cli root span encompassing the entire command lifecycle. In swamp serve and swamp worker, background work and requests do not nest under it; each starts its own trace — see Traces in swamp serve and workers.

Libswamp Generator Spans

Every libswamp generator is wrapped with withGeneratorSpan, producing a span for the full duration of the operation. Generator spans automatically detect kind: "error" events and mark the span status as ERROR.

swamp behavior commands emit the model domain's span names; swamp.model.* span names and model.* attributes are compatibility identifiers and do not change.

Domain-Layer Spans

The workflow execution engine and method execution service create fine-grained spans for the execution hierarchy.

Workflow Evaluation Diagnostics

workflow.expressions_evaluated is the number of distinct raw expressions that evaluated successfully at run start. A successful evaluation also records these diagnostics on the swamp.workflow.evaluate span:

• Repeated occurrences of the same expression still evaluate independently, so workflow.cel_evaluations can exceed workflow.expressions_distinct.
• workflow.model_keys counts every model in the run context, not only the models an expression references.
• Both durations are fractional milliseconds and are 0 when no expression was evaluated. They need not sum to the span's duration, which also covers extraction and substitution.
• A fatal evaluation error ends the span with error status and records none of these diagnostics.
• The diagnostics are counts and durations only. They export no expression text, values or model names.

Infrastructure Spans

Traces in swamp serve and workers

Root spans for background work

Worker dispatches

A dispatch joins the run's trace when serve sends trace headers with it:

swamp.model.method
  └─ swamp.remote.dispatch
       └─ swamp.cli                  ← the runner on the worker
swamp.serve.data_plane               ← one per data-plane read or write, in the run's trace

swamp.serve.data_plane spans carry a swamp.data_plane.operation attribute. Serve takes their trace from its own dispatch record, never from request headers.

Runner environment

• Runners never inherit TRACEPARENT or TRACESTATE from the worker's environment.
• Serve does not send its OTEL_* settings or trace context to workers. A runner exports with the worker host's own OTel settings, so configure OTEL_* on each worker host.
• With OTEL_TRACES_EXPORTER=console, a runner writes its span output to stderr, and it appears in the worker log at debug level.

Export at exit

A process waits up to the exporter's 10-second timeout for in-flight span exports when it exits, on the error path as well as on success. With a slow or unreachable collector, a failing command or a dispatch runner can take up to 10 seconds longer to exit.

Context Propagation

In-Process

The OTel SDK uses AsyncLocalStorageContextManager to propagate trace context through async call chains. tracer.startActiveSpan() and tracer.startSpan() establish parent-child relationships automatically within the active context.

• yield* delegation in async generators preserves context across generator boundaries.

Cross-Process

For out-of-process execution, W3C Trace Context headers are propagated via environment variables.

Per-Invocation

Callers can inject a W3C parent trace context into a single run so it appears as a child of an external trace.

CLI flags — workflow run and behavior run accept --traceparent and --tracestate:

swamp workflow run my-workflow --traceparent "00-<trace-id>-<span-id>-01"
swamp behavior run my-behavior execute --traceparent "00-<trace-id>-<span-id>-01" --tracestate "vendor=value"

The TRACEPARENT and TRACESTATE environment variables provide the same context. When both the flag and the env var are set, the flag takes precedence. An empty env var is treated as unset.

WebSocket protocol — the WorkflowRunPayload and ModelMethodRunPayload messages accept traceparent and tracestate fields, propagating the caller's trace context into the server-side execution.

Webhook header extraction — when a webhook request includes traceparent and/or tracestate HTTP headers, swamp serve propagates them into the triggered workflow execution automatically. No configuration is needed. A webhook run without traceparent starts its own trace; it does not nest under the POST request's span.

Scheduled/cron runs — scheduled runs mint a fresh swamp.scheduled.fire root span as the top of the trace, since there is no external caller to inherit context from.

SDK Lifecycle

initTracing()
  ├─ No endpoint set, exporter not console → return (no-op)
  └─ Endpoint set or console exporter
       ├─ Create AsyncLocalStorageContextManager
       ├─ Create BasicTracerProvider with Resource
       ├─ Add BatchSpanProcessor with OTLP span exporter
       ├─ Register as global tracer provider
       ├─ Create LoggerProvider with Resource
       ├─ Add log record processor with OTLP log exporter
       │    (BatchLogRecordProcessor when OTEL_BLRP_USE=1,
       │     SimpleLogRecordProcessor otherwise)
       └─ Register as global logger provider

runCli(args)          ← spans and log records created during execution

shutdownTracing()     ← flush pending spans and log records, disable providers

All OTel SDK packages (sdk-trace-base, sdk-logs, otlp-transformer, context-async-hooks, resources, semantic-conventions) are loaded via dynamic import() only when telemetry is enabled. @opentelemetry/api and @opentelemetry/api-logs are statically imported — they return no-op implementations when no provider is registered.

Both OTLP exporters (spans and logs) use Deno's native fetch API instead of the Node.js http/https modules. Spans POST to /v1/traces and logs POST to /v1/logs on the configured endpoint. @opentelemetry/otlp-transformer handles JSON serialization for both signals.

Utilities

withSpan(name, attributes, fn)

Wraps an async function with a span. Creates the span, sets it as active, executes the function, records errors, and ends the span in all code paths.

await withSpan("swamp.workflow.run", { "workflow.name": name }, async () => {
  // span is active for the duration
});

withGeneratorSpan(name, attributes, generator)

Wraps an async generator with a span. The span starts when iteration begins and ends when the generator completes, throws, or is abandoned. Events with kind: "error" are detected and recorded as span errors.

yield * withGeneratorSpan("swamp.data.get", { "data.name": name }, getData());

getTracer()

Returns the Swamp tracer from the global provider. Returns a no-op tracer when tracing is not initialized.

injectTraceContext()

Extracts W3C Trace Context headers (traceparent, tracestate) from the current active context into a Record<string, string>. Returns an empty object when tracing is disabled.

extractTraceContext(headers)

Creates a context from W3C Trace Context headers for connecting incoming spans to an existing trace.

withGeneratorTraceContext(traceparent, tracestate, generator)

Wraps an async generator so it runs within the trace context specified by the given traceparent and tracestate values. Used internally to attach a per-invocation parent context to workflow and behavior method executions.

Behavior When Disabled

When telemetry is disabled: