Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4511 Observability never slows or breaks a command: the span exporter caps its connections, and no command waits on telemetry or pays slower start-up (F36, F33 from #4374)

Opened by stack72 · 11/8/2026· Shipped 11/8/2026

Found by the team-mode dogfood on real GCS (swamp-club#4374), run 4, release 20261106.094050.0-sha.fcdd7e60. This is group 5 of the dogfood fixes. It blocks dogfood run 5: while it's open, run 5's measurements can't be trusted. One owner, one PR. It runs in parallel with the F37 issue (the runaway read-model rewrite).

The principle: observability must never slow down or break the command it observes. Run 4 broke it in two ways.

1. F36: the span exporter exhausts file descriptors and fails workflows (regression)

What run 4 saw:

  • Members export spans with OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 to a local single-threaded collector, as in run 3.
  • workflow run nightly-report succeeded 157 of 2,594 times (6%) across run 4's load and a 25-hour soak. Run 3, with the same collector and the same load, succeeded 2,259 of 2,424 (93%).
  • Almost every failure is Too many open files (os error 24), on a writefile under .swamp/outputs/ or a readdir under .swamp/data/.
  • A live sample: one swamp process went from 38 to 1,024 file descriptors in about a second, 993 of them sockets to the collector. Another showed 134 established and 85 still connecting to 127.0.0.1, and only 3 to serve.
  • The 1,000-step forEach failed with the same error after 979 s, in run 3 and run 4 (F13).

Why, in the code:

  • The CLI defaults to SimpleSpanProcessor (src/infrastructure/tracing/otel_init.ts, "Deno.exit can beat BatchSpanProcessor's window"). So every span is exported the moment it ends, as its own fetch.
  • FetchOtlpExporter (src/infrastructure/tracing/fetch_otlp_exporter.ts) tracks in-flight sends for forceFlush, but places no limit on how many are in flight.
  • With a collector that answers slowly, each span that ends opens another connection while the earlier ones wait. File descriptors climb until the command's own file I/O fails.
  • The design isn't new. #4457 likely tipped it over by adding spans that end on every step: swamp.method.process, lock_hand_off and lock_hand_back, arguments_validated, datastore.v3.commit.append_group.

Fix direction:

  • The exporter keeps at most a small fixed number of sends in flight (one or two). Spans that end while a send is in flight join the next batch, so one request carries many spans.
  • Past a fixed queue limit, drop spans and count the drops in the existing swamp.export.spans counter. Never block, and never fail the command.
  • On exit, drain what's queued within a short bound, then give up. Exit stays fast, which is why SimpleSpanProcessor was chosen.
  • Serve already opts into batching (OTEL_BSP_USE=1). Check the same limit applies there too.

Tests:

  • A collector that accepts connections and never answers: a many-step workflow completes, its descriptor count stays bounded (compare against a fixed limit, not a timing), and the command exits within the drain bound.
  • A slow collector: spans still arrive, batched, with no more than the in-flight limit open at once.
  • A collector that refuses connections: the command is unaffected.

2. F33: commands still wait on telemetry, and start-up got slower

What run 4 saw (swamp version, 10 runs each, near and far members, Linux x86_64):

  • Telemetry still costs time: 2.7–4.4 s with telemetry on against 1.8–2.9 s off, so about 0.7–1.5 s.
    • #4457 meant to remove that wait with the detached sender (macOS and Linux); on these Linux VMs it doesn't hold.
  • Start-up with telemetry off is slower: about 2.2–2.6 s, against about 1.6 s on run 3's release (20261105.030003.0). So start-up grew by about 0.6–1.0 s, before any telemetry.
  • On the dogfood VMs, OTel export was on (item 1), so measure with it off as well.

What to do:

  • Measure first, on Linux:
    • swamp version on run 3's release, on 20261105.234357.0 (before #4457) and on run 4's release;
    • telemetry on and off, OTel export off;
    • a container is fine.
  • For the telemetry cost:
    • Check whether the sender really detaches on Linux. Is the spawn awaited? Does the parent wait on an inherited pipe? Does a stdio handle keep the process alive?
    • Check what else waits at the command's end: telemetry set-up, auth file reads, the spool write's fsync.
    • The bar: telemetry on costs no more than a few tens of milliseconds over off.
  • For the start-up growth:
    • Find what the releases since run 3 added before the first span (module loading, the auth gate, config reads), and cut it, or report what it is.
    • The bar: back to run 3's release or better.

Tests:

  • A test where the telemetry ingest never answers: the command's exit isn't delayed (assert on work done or process state, not wall-clock bounds).
  • The detached-sender tests from #4457 pass on Linux in CI.

Must not regress

  • Spans still reach a healthy collector, and serve's spans still join the member's trace (F25, 100% in run 4).
  • Telemetry entries still reach the ingest; swamp-uat telemetry/stats_test.ts passes unchanged.
  • Windows keeps its awaited telemetry send (#4457).

Done when

  • Each fix has a test that fails on current main.
  • On a compiled Linux binary:
    • with a never-answering collector, a 50-step workflow completes, with a bounded descriptor count;
    • swamp version with telemetry on is within a few tens of ms of off;
    • start-up is back at run 3's level, or the remaining cost is reported.
  • Dogfood run 5 then verifies on GCS: an OTel on/off A/B and the workflow success rate over a soak (the bar is run 3's 93% or better).
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 7 MOREREVIEW+ 13 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/8/2026, 2:21:07 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/8/2026, 12:03:03 AM

Sign in to post a ripple.