Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4640 workflow run: every save converts the whole run record to YAML with @std/yaml, the largest remaining cost of a long run

Opened by zack · 11/11/2026· Shipped 11/11/2026

Description

Every save of a workflow run converts the run's whole record to YAML with @std/yaml stringify (YamlWorkflowRunRepository.save() in src/infrastructure/persistence/yaml_workflow_run_repository.ts). A run saves twice per step and its record grows by about 9 KB of YAML per step, so the total cost grows with the square of the step count.

After swamp-club#4638 removed the read-back before each save, this is the largest remaining cost of a long run.

Impact

Measured on the fixed build (after #4638), with serial command/shell echo steps and a fresh repo per run:

  • Run times: 585 ms (10 steps), 918 ms (25), 1.65 s (50), 3.78 s (100). Each extra step costs more as the run gets longer: 22, 29, then 43 ms.
  • Share of the run: in a V8 tick profile of a 100-step run, @std/yaml stringify is 38.9% of all samples. Almost all of it comes from the run repository's save.
  • Per save: 1.76 ms with 10 steps in the record, 6.06 ms with 50, 12.22 ms with 100.
  • Where the time goes: the hottest function is chooseScalarStyle, which for every string scans characters for plain-safety, measures line lengths for folding, and tests the string against the int/float/bool/null/timestamp patterns. A separate full pass (getDuplicateObjects) looks for anchors.

The other per-save costs are small at 100 steps: about 120 ms for the JSON clone, about 45 ms for the fingerprintIfContent read and about 30 ms for the atomic write.

Proposed fix

Replace stringifyYaml in save() with a block-style YAML writer specialised for the record. The record is already JSON-clean (JSON.parse(JSON.stringify(data)) runs just before the stringify), so the writer can skip YAML's style choices:

  • A string is written plain only when it is trivially safe: it starts with a letter, _ or /, contains only [A-Za-z0-9_./@-], and is not a reserved word (true, false, null, yes, no, on, off, y, n, in any case).
  • Every other string is written with V8's native JSON.stringify(str). A JSON string literal is a valid YAML 1.2 double-quoted scalar.
  • Keys follow the same rule. Numbers, booleans and null are written as JSON writes them, and empty objects and arrays as {} and [].
  • No folding, no anchors, no type guessing.

The output is still ordinary block-style YAML:

workflowName: bench100
status: succeeded
startedAt: "2026-11-11T14:39:13.716Z"
jobs:
- jobName: main
  steps:
  - stepName: s1

All readers stay on @std/yaml parse, so older binaries, serve and the forwarded-content authorization keep reading new records, and the new code keeps reading old ones.

A prototype of about 60 lines, measured on a real 100-step record (887 KB):

Writer Time Output size
@std/yaml stringify (today) 12.3 ms 887 KB
npm yaml stringify 59.1 ms 883 KB
Specialised writer (prototype) 3.0 ms 762 KB
JSON.stringify(data, null, 2) (reference) 1.4 ms 1,116 KB
  • Correctness: @std/yaml parses the prototype's output back to data identical to the input.
  • Faster reads too: reading the new output with the existing @std/yaml parse is about 35% faster (24.5 ms against 37.7 ms), because quoted strings skip type resolution.
  • Estimated effect: about 1.2 s down to about 0.3 s of YAML work in a 100-step run.

Must be covered before shipping

  • Characters JSON leaves unescaped but YAML forbids in a stream: DEL and the C1 controls (U+007F to U+009F) must be escaped as \uXXXX. Check how @std/yaml treats U+2028/U+2029 and lone surrogates.
  • A fast-check property test: for any JSON-clean value, parse(write(value)) deep-equals value. Cover empty strings, keys such as <<, ~ and -, number-like and timestamp-like strings, leading and trailing spaces, newlines, unicode, empty objects and arrays inside arrays, and deep nesting (build expected values without recursion; Windows has a shallow stack).
  • Text-based assumptions: check for anything that inspects the record text rather than parsing it. Damage detection relies on a whole write ending in a newline, so the writer must keep the trailing newline. Byte comparisons such as the unchanged check and the v3 digests are unaffected by format.
  • Other writers: decide whether other run-record writers should switch too, so one record never mixes two writers in one release.

Alternatives considered

  1. JSON format in the same .yaml file, with a JSON.parse fast path in parseRecord. JSON is valid YAML, so existing readers still work. It is fastest (writes 1.4 ms, reads 0.9 ms with the fast path), but files are 26% larger and harder to read. Worth reconsidering if read cost becomes the issue.
  2. Slimming the record. About 90% of each step's bytes are repetition: dataArtifacts is stored twice (on the step and in step.output), the 11-key tag block appears 5 times for a resource and 4 times for a file in every step, and dataHandles embeds metadata the data store already holds. This compounds with the writer change, but needs a trace of every reader of these fields first (CEL steps.<name>.output, history get --json, nested runs, gc and retention). Better as its own issue.
  3. Caching each settled step's YAML and stitching the file together. It keeps today's exact style, but folding differs by indent, it relies on settled steps never changing, it is more code, and reads gain nothing.
  4. A native YAML library over FFI (Rust) or WebAssembly. Rejected:
    • JS objects cannot cross FFI, so each save still pays JSON.stringify plus a copy each way, about the same as the specialised writer.
    • It would mean shipping a native library for each platform and extracting it to disk for dlopen.
    • Rust YAML crates resolve types differently from @std/yaml's YAML 1.2 core schema (YAML 1.1 reads yes, no, on and 0777 as booleans or numbers), and serde_yaml is unmaintained.
  5. Tuning @std/yaml options. lineWidth: -1 or disabling anchors gives only 20 to 25% (16.7 ms down to 12.5 to 13.4 ms).

Steps to reproduce

  1. swamp repo init --tool none --no-vault, then create a command/shell behavior sh1 whose execute method runs echo bench.
  2. Create workflows of 10, 25, 50 and 100 serial model_method steps calling sh1 / execute, each depending on the previous one.
  3. Run each in a fresh repo and time it: the cost per step rises with the step count.
  4. Profile with deno run --v8-flags=--prof, then node --prof-process. The @std/yaml dumper (chooseScalarStyle, stringifyNode, stringifyScalar) dominates.

Environment

Current main (after #4638), Linux x86_64, Deno 2.9.8, @std/yaml 1.1.0.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 11 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/11/2026, 4:13:36 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/11/2026, 2:57:31 PM

Sign in to post a ripple.