Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneeszack

Relationships

≡ duplicated by #4301

#4285 Worker set_status writes an output record on every dispatch, so outputs grow without bound

Opened by nckrtl · 11/2/2026· Shipped 11/2/2026

Summary

Each worker status change runs as a full swamp/worker set_status method run, and every method run writes an output record. A dispatch records busy and then idle, so every dispatch writes two outputs under .swamp/outputs/swamp/worker/set_status/. The state resource sets garbageCollection: 20, so its data is bounded, but the outputs are bounded only by the outputs retention window. On a busy serve they pile up to tens of thousands of files.

Version

swamp 20261031.063736.0-sha.e459d219, swamp serve --remote-only --hot-reload --auth-mode token, filesystem datastore, Linux x86_64. Two workers with --concurrency 8.

Measurements (one serve, 2026-11-02)

  • .swamp/outputs/swamp/worker/set_status/ held 24,215 files, growing by about 44 per minute.
  • A 3-second strace of serve showed 833 realpath-style walks, each about 12 readlink calls. 48% of them resolved paths under that directory.

Code path

WorkerGateway dispatch calls #recordTransition("worker_busy", () => this.#runModelMethod({ typeArg: WORKER_MODEL_TYPE.normalized, definitionName: workerDefinitionName(name), methodName: "set_status", inputs: { status: "busy", activeDispatchIds } })), and the matching idle transition on completion. Both go through the normal method run path, which saves an output record.

Expected

A status transition costs no output record, or its outputs are bounded like the resource data (keep the last N). Status is control-plane bookkeeping: nobody reads a set_status output, and its count should not grow with the number of dispatches.

Reproduce

  1. Run swamp serve with one worker connected.
  2. Dispatch a few hundred short steps, for example a workflow that runs true in a forEach.
  3. Count the files under .swamp/outputs/swamp/worker/set_status/: about two per dispatch.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 12 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/2/2026, 6:30:21 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/2/2026, 3:28:28 PM
zack linked duplicated by #430111/20/2026, 8:24:54 PM
Editable. Press Enter to edit.

nckrtl commented 11/2/2026, 1:45:09 PM

More numbers from the same serve, in case they help. We only have totals for the whole outputs/ directory, not the set_status share, so please read this as context rather than proof.

Setup: the same e459 build and filesystem datastore, with a fresh datastore since 07:37Z. Serve ran at about 50 dispatches a minute.

What we observed:

  • From 13:00Z to 13:15Z, outputs/ grew from 60,515 to 64,855 files. That is about 4,300 in 15 minutes, roughly 17,000 an hour.
  • At about 3,000 dispatches an hour, that is around 5 to 6 output files per dispatch across all methods. The two set_status runs per dispatch would be part of that.
  • With outputs: 1d retention, serve's 13:07Z retention pass removed 0 outputs, because none were old enough yet. At this rate the directory would reach about 400,000 files before the first ones age out.
  • top -H on serve showed nearly all CPU time on the main thread: 24:07 of about 27 minutes. The four V8 worker threads had about 0:53 each. CPU came in bursts that follow dispatch volume, 80 to 115% at peaks and 0.4% when quiet.

What we concluded (we may be wrong): per-dispatch output writes look like a large share of datastore growth on a busy serve. Whether they also drive the main-thread bursts we can't say without a profile.

What we did: we set garbageCollection.outputs: 6h as an interim measure. If you'd suggest a better setting, we'd welcome it.

nckrtl commented 11/2/2026, 2:46:49 PM

One more observation. It is a lead, not a conclusion.

What we saw: on the same serve, between 12:30Z and 14:30Z:

  • The dispatch rate roughly halved, judged by step-lease growth per 15 minutes: about 5,000 to 5,700 before 13:00Z, about 2,100 to 3,600 after.
  • Serve's CPU did not drop. Five-second samples averaged about 51% in both windows. They were bursty: near 0%, or 50 to 95%.
  • That means each dispatch cost roughly twice as much CPU at 14:30Z as at noon.
  • The only store that grew steadily over the same time was outputs/, from 52,736 to 78,368 files. Step leases rose and fell with the hourly worker GC sweep, and auto-definitions held at about 600.
  • From an earlier top -H: nearly all CPU time is on serve's main thread.

Why we mention it: the strace in the description found 48% of path walks resolving paths under outputs/swamp/worker/set_status/. If something in the dispatch path touches that directory in a way that scales with its size, per-dispatch cost would grow as outputs accumulate. That would fit what we see.

Caveats:

  • Five-second CPU samples are noisy, and we have not profiled serve.
  • The workload mix may also have changed between the two windows.
  • We may well be wrong about the mechanism.

What we are trying: garbageCollection.outputs: 2h, which caps outputs at about 35,000 at our rate. We'll report whether per-dispatch CPU stops rising once outputs level off.

nckrtl commented 11/2/2026, 6:16:34 PM

A follow-up on our earlier lead, which the data now seems to contradict.

What we changed: garbageCollection.outputs went from 1d to 2h (live at about 15:07Z), then to 1h (live at about 17:08Z). Outputs dropped from about 78,000 files to between 13,000 and 23,000.

What we observed after (17:12Z to 18:15Z, five-second CPU samples): while dispatches ran at about the same rate as at noon (step-lease growth of 5,000 to 6,000 per 15 minutes), serve's CPU samples were 35%, 15%, 91% and 80%. That is about the level we saw at noon with roughly 50,000 to 78,000 outputs.

What we now think (we may still be wrong): the number of output files does not seem to drive the per-dispatch CPU cost, so please set aside our earlier lead. Short retention still keeps the directory small, which is useful on its own. Serve's CPU looks like it tracks dispatch volume on the main thread, in bursts, rather than the size of outputs/.

zack commented 11/2/2026, 6:30:23 PM

Thanks @nckrtl for reporting this! We shipped: Worker status transitions stop running as full swamp/worker set_status method runs. A new WorkerStatusLedger in src/serve, modelled on StepLeaseLedger (swamp-club#4038), reads the worker state record, applies the same pure transition the method uses, and writes the next version directly. A dispatch then costs two state versions (kept to the last 20 by swamp data gc) and no output record, run log or evaluated-definition save. The method stays the interface for swamp behavior run, and gains outputLifetime 1d so the existing backlog can be pruned.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.