Relationships
#4640 workflow run: every save converts the whole run record to YAML with @std/yaml, the largest remaining cost of a long run
Opened by zack · 11/11/2026· Shipped 11/11/2026
Description
Every save of a workflow run converts the run's whole record to YAML with @std/yaml stringify (YamlWorkflowRunRepository.save() in src/infrastructure/persistence/yaml_workflow_run_repository.ts). A run saves twice per step and its record grows by about 9 KB of YAML per step, so the total cost grows with the square of the step count.
After swamp-club#4638 removed the read-back before each save, this is the largest remaining cost of a long run.
Impact
Measured on the fixed build (after #4638), with serial command/shell echo steps and a fresh repo per run:
- Run times: 585 ms (10 steps), 918 ms (25), 1.65 s (50), 3.78 s (100). Each extra step costs more as the run gets longer: 22, 29, then 43 ms.
- Share of the run: in a V8 tick profile of a 100-step run,
@std/yamlstringify is 38.9% of all samples. Almost all of it comes from the run repository's save. - Per save: 1.76 ms with 10 steps in the record, 6.06 ms with 50, 12.22 ms with 100.
- Where the time goes: the hottest function is
chooseScalarStyle, which for every string scans characters for plain-safety, measures line lengths for folding, and tests the string against the int/float/bool/null/timestamp patterns. A separate full pass (getDuplicateObjects) looks for anchors.
The other per-save costs are small at 100 steps: about 120 ms for the JSON clone, about 45 ms for the fingerprintIfContent read and about 30 ms for the atomic write.
Proposed fix
Replace stringifyYaml in save() with a block-style YAML writer specialised for the record. The record is already JSON-clean (JSON.parse(JSON.stringify(data)) runs just before the stringify), so the writer can skip YAML's style choices:
- A string is written plain only when it is trivially safe: it starts with a letter,
_or/, contains only[A-Za-z0-9_./@-], and is not a reserved word (true,false,null,yes,no,on,off,y,n, in any case). - Every other string is written with V8's native
JSON.stringify(str). A JSON string literal is a valid YAML 1.2 double-quoted scalar. - Keys follow the same rule. Numbers, booleans and null are written as JSON writes them, and empty objects and arrays as
{}and[]. - No folding, no anchors, no type guessing.
The output is still ordinary block-style YAML:
workflowName: bench100
status: succeeded
startedAt: "2026-11-11T14:39:13.716Z"
jobs:
- jobName: main
steps:
- stepName: s1All readers stay on @std/yaml parse, so older binaries, serve and the forwarded-content authorization keep reading new records, and the new code keeps reading old ones.
A prototype of about 60 lines, measured on a real 100-step record (887 KB):
| Writer | Time | Output size |
|---|---|---|
@std/yaml stringify (today) |
12.3 ms | 887 KB |
npm yaml stringify |
59.1 ms | 883 KB |
| Specialised writer (prototype) | 3.0 ms | 762 KB |
JSON.stringify(data, null, 2) (reference) |
1.4 ms | 1,116 KB |
- Correctness:
@std/yamlparses the prototype's output back to data identical to the input. - Faster reads too: reading the new output with the existing
@std/yamlparse is about 35% faster (24.5 ms against 37.7 ms), because quoted strings skip type resolution. - Estimated effect: about 1.2 s down to about 0.3 s of YAML work in a 100-step run.
Must be covered before shipping
- Characters JSON leaves unescaped but YAML forbids in a stream: DEL and the C1 controls (U+007F to U+009F) must be escaped as
\uXXXX. Check how@std/yamltreats U+2028/U+2029 and lone surrogates. - A fast-check property test: for any JSON-clean value,
parse(write(value))deep-equalsvalue. Cover empty strings, keys such as<<,~and-, number-like and timestamp-like strings, leading and trailing spaces, newlines, unicode, empty objects and arrays inside arrays, and deep nesting (build expected values without recursion; Windows has a shallow stack). - Text-based assumptions: check for anything that inspects the record text rather than parsing it. Damage detection relies on a whole write ending in a newline, so the writer must keep the trailing newline. Byte comparisons such as the
unchangedcheck and the v3 digests are unaffected by format. - Other writers: decide whether other run-record writers should switch too, so one record never mixes two writers in one release.
Alternatives considered
- JSON format in the same
.yamlfile, with aJSON.parsefast path inparseRecord. JSON is valid YAML, so existing readers still work. It is fastest (writes 1.4 ms, reads 0.9 ms with the fast path), but files are 26% larger and harder to read. Worth reconsidering if read cost becomes the issue. - Slimming the record. About 90% of each step's bytes are repetition:
dataArtifactsis stored twice (on the step and instep.output), the 11-key tag block appears 5 times for a resource and 4 times for a file in every step, anddataHandlesembeds metadata the data store already holds. This compounds with the writer change, but needs a trace of every reader of these fields first (CELsteps.<name>.output,history get --json, nested runs, gc and retention). Better as its own issue. - Caching each settled step's YAML and stitching the file together. It keeps today's exact style, but folding differs by indent, it relies on settled steps never changing, it is more code, and reads gain nothing.
- A native YAML library over FFI (Rust) or WebAssembly. Rejected:
- JS objects cannot cross FFI, so each save still pays
JSON.stringifyplus a copy each way, about the same as the specialised writer. - It would mean shipping a native library for each platform and extracting it to disk for
dlopen. - Rust YAML crates resolve types differently from
@std/yaml's YAML 1.2 core schema (YAML 1.1 readsyes,no,onand0777as booleans or numbers), andserde_yamlis unmaintained.
- JS objects cannot cross FFI, so each save still pays
- Tuning
@std/yamloptions.lineWidth: -1or disabling anchors gives only 20 to 25% (16.7 ms down to 12.5 to 13.4 ms).
Steps to reproduce
swamp repo init --tool none --no-vault, then create acommand/shellbehaviorsh1whoseexecutemethod runsecho bench.- Create workflows of 10, 25, 50 and 100 serial
model_methodsteps callingsh1/execute, each depending on the previous one. - Run each in a fresh repo and time it: the cost per step rises with the step count.
- Profile with
deno run --v8-flags=--prof, thennode --prof-process. The@std/yamldumper (chooseScalarStyle,stringifyNode,stringifyScalar) dominates.
Environment
Current main (after #4638), Linux x86_64, Deno 2.9.8, @std/yaml 1.1.0.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.