OPENTELEMETRY

Swamp uses OpenTelemetry (OTel) for distributed tracing and structured log export. Both signals are opt-in and controlled entirely through standard OTel environment variables. When no provider is registered, all tracer, span, and logger operations are no-ops with zero overhead.

Configuration

Telemetry is enabled when OTEL_EXPORTER_OTLP_ENDPOINT is set or either OTEL_TRACES_EXPORTER or OTEL_LOGS_EXPORTER is console. When neither condition is met, no SDK packages are loaded and no async hooks are installed.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 swamp workflow run my-workflow

Setting OTEL_TRACES_EXPORTER=console prints spans to stderr without requiring a collector endpoint. Setting OTEL_LOGS_EXPORTER=console prints log records to stderr in the same way.

Logs

Structured log lines emitted during a run are exported as native OTLP log records, sent to the collector's /v1/logs endpoint. Logs share the same OTEL_EXPORTER_OTLP_ENDPOINT as traces — when the endpoint is set, both signals are active by default.

Trace Correlation

Each log record carries the trace_id and span_id of the active span at the time the log was emitted. This links log lines to the span that produced them, so a collector or backend can display logs inline with their parent trace.

Redaction

Run secrets (vault values, environment variables marked sensitive) are redacted from log record bodies and attributes before export. Redaction happens in the export pipeline, so secrets never leave the process in OTLP payloads.

Coverage

Log export covers swamp serve and worker subprocesses. Worker subprocesses receive trace context via the propagated TRACEPARENT environment variable, so their log records are correlated with the parent trace.

Batching

By default, log records are exported individually (one OTLP request per record). For long-running processes like swamp serve, set OTEL_BLRP_USE=1 to enable the BatchLogRecordProcessor, which buffers records and flushes them periodically — reducing request volume to the collector.

Span Naming Convention

All spans use the prefix swamp. followed by the domain and operation:

swamp.<domain>.<operation>

Span Hierarchy

A workflow run produces a trace with the following structure:

swamp.cli
  ├─ swamp.lock.acquire
  ├─ swamp.datastore.sync
  └─ swamp.workflow.run.command
       └─ swamp.workflow.run
            ├─ swamp.workflow.evaluate
            ├─ swamp.workflow.job
            │    ├─ swamp.workflow.step
            │    │    └─ swamp.model.method
            │    └─ swamp.workflow.step
            │         └─ swamp.model.method
            └─ swamp.workflow.job
                 └─ swamp.workflow.step
                      └─ swamp.model.method

A scheduled/cron run creates a separate root span:

swamp.scheduled.fire
  └─ swamp.workflow.run
       └─ ...

CLI Root Span

Every CLI invocation creates a swamp.cli root span encompassing the entire command lifecycle.

Libswamp Generator Spans

Every libswamp generator is wrapped with withGeneratorSpan, producing a span for the full duration of the operation. Generator spans automatically detect kind: "error" events and mark the span status as ERROR.

Domain-Layer Spans

The workflow execution engine and method execution service create fine-grained spans for the execution hierarchy.

Infrastructure Spans

Context Propagation

In-Process

The OTel SDK uses AsyncLocalStorageContextManager to propagate trace context through async call chains. tracer.startActiveSpan() and tracer.startSpan() establish parent-child relationships automatically within the active context.

• yield* delegation in async generators preserves context across generator boundaries.

Cross-Process

For out-of-process execution, W3C Trace Context headers are propagated via environment variables.

Per-Invocation

Callers can inject a W3C parent trace context into a single run so it appears as a child of an external trace.

CLI flags — workflow run and model method run accept --traceparent and --tracestate:

swamp workflow run my-workflow --traceparent "00-<trace-id>-<span-id>-01"
swamp model method run my-model execute --traceparent "00-<trace-id>-<span-id>-01" --tracestate "vendor=value"

The TRACEPARENT and TRACESTATE environment variables provide the same context. When both the flag and the env var are set, the flag takes precedence. An empty env var is treated as unset.

WebSocket protocol — the WorkflowRunPayload and ModelMethodRunPayload messages accept traceparent and tracestate fields, propagating the caller's trace context into the server-side execution.

Webhook header extraction — when a webhook request includes traceparent and/or tracestate HTTP headers, swamp serve propagates them into the triggered workflow execution automatically. No configuration is needed.

Scheduled/cron runs — scheduled runs mint a fresh swamp.scheduled.fire root span as the top of the trace, since there is no external caller to inherit context from.

SDK Lifecycle

initTracing()
  ├─ No endpoint set, exporter not console → return (no-op)
  └─ Endpoint set or console exporter
       ├─ Create AsyncLocalStorageContextManager
       ├─ Create BasicTracerProvider with Resource
       ├─ Add BatchSpanProcessor with OTLP span exporter
       ├─ Register as global tracer provider
       ├─ Create LoggerProvider with Resource
       ├─ Add log record processor with OTLP log exporter
       │    (BatchLogRecordProcessor when OTEL_BLRP_USE=1,
       │     SimpleLogRecordProcessor otherwise)
       └─ Register as global logger provider

runCli(args)          ← spans and log records created during execution

shutdownTracing()     ← flush pending spans and log records, disable providers

All OTel SDK packages (sdk-trace-base, sdk-logs, otlp-transformer, context-async-hooks, resources, semantic-conventions) are loaded via dynamic import() only when telemetry is enabled. @opentelemetry/api and @opentelemetry/api-logs are statically imported — they return no-op implementations when no provider is registered.

Both OTLP exporters (spans and logs) use Deno's native fetch API instead of the Node.js http/https modules. Spans POST to /v1/traces and logs POST to /v1/logs on the configured endpoint. @opentelemetry/otlp-transformer handles JSON serialization for both signals.

Utilities

withSpan(name, attributes, fn)

Wraps an async function with a span. Creates the span, sets it as active, executes the function, records errors, and ends the span in all code paths.

await withSpan("swamp.workflow.run", { "workflow.name": name }, async () => {
  // span is active for the duration
});

withGeneratorSpan(name, attributes, generator)

Wraps an async generator with a span. The span starts when iteration begins and ends when the generator completes, throws, or is abandoned. Events with kind: "error" are detected and recorded as span errors.

yield * withGeneratorSpan("swamp.data.get", { "data.name": name }, getData());

getTracer()

Returns the Swamp tracer from the global provider. Returns a no-op tracer when tracing is not initialized.

injectTraceContext()

Extracts W3C Trace Context headers (traceparent, tracestate) from the current active context into a Record<string, string>. Returns an empty object when tracing is disabled.

extractTraceContext(headers)

Creates a context from W3C Trace Context headers for connecting incoming spans to an existing trace.

withGeneratorTraceContext(traceparent, tracestate, generator)

Wraps an async generator so it runs within the trace context specified by the given traceparent and tracestate values. Used internally to attach a per-invocation parent context to workflow and model method executions.

Behavior When Disabled

When telemetry is disabled: