← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72
Relationships
#4511 Observability never slows or breaks a command: the span exporter caps its connections, and no command waits on telemetry or pays slower start-up (F36, F33 from #4374)
Opened by stack72 · 11/8/2026· Shipped 11/8/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), run 4, release 20261106.094050.0-sha.fcdd7e60. This is group 5 of the dogfood fixes. It blocks dogfood run 5: while it's open, run 5's measurements can't be trusted. One owner, one PR. It runs in parallel with the F37 issue (the runaway read-model rewrite).
The principle: observability must never slow down or break the command it observes. Run 4 broke it in two ways.
1. F36: the span exporter exhausts file descriptors and fails workflows (regression)
What run 4 saw:
- Members export spans with
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318to a local single-threaded collector, as in run 3. workflow run nightly-reportsucceeded 157 of 2,594 times (6%) across run 4's load and a 25-hour soak. Run 3, with the same collector and the same load, succeeded 2,259 of 2,424 (93%).- Almost every failure is
Too many open files (os error 24), on awritefileunder.swamp/outputs/or areaddirunder.swamp/data/. - A live sample: one swamp process went from 38 to 1,024 file descriptors in about a second, 993 of them sockets to the collector. Another showed 134 established and 85 still connecting to
127.0.0.1, and only 3 to serve. - The 1,000-step
forEachfailed with the same error after 979 s, in run 3 and run 4 (F13).
Why, in the code:
- The CLI defaults to
SimpleSpanProcessor(src/infrastructure/tracing/otel_init.ts, "Deno.exit can beat BatchSpanProcessor's window"). So every span is exported the moment it ends, as its ownfetch. FetchOtlpExporter(src/infrastructure/tracing/fetch_otlp_exporter.ts) tracks in-flight sends forforceFlush, but places no limit on how many are in flight.- With a collector that answers slowly, each span that ends opens another connection while the earlier ones wait. File descriptors climb until the command's own file I/O fails.
- The design isn't new. #4457 likely tipped it over by adding spans that end on every step:
swamp.method.process,lock_hand_offandlock_hand_back,arguments_validated,datastore.v3.commit.append_group.
Fix direction:
- The exporter keeps at most a small fixed number of sends in flight (one or two). Spans that end while a send is in flight join the next batch, so one request carries many spans.
- Past a fixed queue limit, drop spans and count the drops in the existing
swamp.export.spanscounter. Never block, and never fail the command. - On exit, drain what's queued within a short bound, then give up. Exit stays fast, which is why
SimpleSpanProcessorwas chosen. - Serve already opts into batching (
OTEL_BSP_USE=1). Check the same limit applies there too.
Tests:
- A collector that accepts connections and never answers: a many-step workflow completes, its descriptor count stays bounded (compare against a fixed limit, not a timing), and the command exits within the drain bound.
- A slow collector: spans still arrive, batched, with no more than the in-flight limit open at once.
- A collector that refuses connections: the command is unaffected.
2. F33: commands still wait on telemetry, and start-up got slower
What run 4 saw (swamp version, 10 runs each, near and far members, Linux x86_64):
- Telemetry still costs time: 2.7–4.4 s with telemetry on against 1.8–2.9 s off, so about 0.7–1.5 s.
- #4457 meant to remove that wait with the detached sender (macOS and Linux); on these Linux VMs it doesn't hold.
- Start-up with telemetry off is slower: about 2.2–2.6 s, against about 1.6 s on run 3's release (
20261105.030003.0). So start-up grew by about 0.6–1.0 s, before any telemetry. - On the dogfood VMs, OTel export was on (item 1), so measure with it off as well.
What to do:
- Measure first, on Linux:
swamp versionon run 3's release, on20261105.234357.0(before #4457) and on run 4's release;- telemetry on and off, OTel export off;
- a container is fine.
- For the telemetry cost:
- Check whether the sender really detaches on Linux. Is the spawn awaited? Does the parent wait on an inherited pipe? Does a stdio handle keep the process alive?
- Check what else waits at the command's end: telemetry set-up, auth file reads, the spool write's fsync.
- The bar: telemetry on costs no more than a few tens of milliseconds over off.
- For the start-up growth:
- Find what the releases since run 3 added before the first span (module loading, the auth gate, config reads), and cut it, or report what it is.
- The bar: back to run 3's release or better.
Tests:
- A test where the telemetry ingest never answers: the command's exit isn't delayed (assert on work done or process state, not wall-clock bounds).
- The detached-sender tests from #4457 pass on Linux in CI.
Must not regress
- Spans still reach a healthy collector, and serve's spans still join the member's trace (F25, 100% in run 4).
- Telemetry entries still reach the ingest; swamp-uat
telemetry/stats_test.tspasses unchanged. - Windows keeps its awaited telemetry send (#4457).
Done when
- Each fix has a test that fails on current main.
- On a compiled Linux binary:
- with a never-answering collector, a 50-step workflow completes, with a bounded descriptor count;
swamp versionwith telemetry on is within a few tens of ms of off;- start-up is back at run 3's level, or the remaining cost is reported.
- Dogfood run 5 then verifies on GCS: an OTel on/off A/B and the workflow success rate over a soak (the bar is run 3's 93% or better).
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.