Full run-index rewrites stall a shared serve event loop as history grows

Swamp version: official pinned e459d219, Linux.

Each workflow-run save reads and parses the workflow’s entire .runs-index.json, updates one entry, then synchronously serializes and rewrites the complete index. Every entry includes all immutable run inputs. A metadata-capture workflow receiving about 155 KB of public inputs reached 885 entries and a 136.8 MB index in under six hours.

On the serve host, a separate Deno process measured an identical read / parse / stringify / write / rename cycle against a scratch copy, without changing the live index:

• Read: 581 ms.
• JSON.parse: 522 ms.
• JSON.stringify: 1,004 ms.
• Write: 739 ms.
• Rename: 0.315 ms.
• Total: 2,847 ms for 136,772,927 bytes and 885 entries.

Repeated serialization trials took 0.39–0.62 ms for one entry, about 4.6 ms for 10 entries, 73–123 ms for 100 entries, and 561 / 3,401 / 684 ms for 877 entries. Allocation, garbage collection and host contention can affect individual trials. A minimal reproduction using only dummy inputs against the pinned source on a separate host measured a complete first-write / read-parse / second-write cycle of 8.7 ms for one entry, 151 ms for 100 entries and 1,803 ms for 875 entries.

A 25-second strace of the live serve process, with string capture disabled, observed repeated 135,213,613-byte index writes. The largest individual write was 94.7 ms, fsync 236 ms and rename 257 ms. The multi-second index cost is not one long disk syscall.

The update queue is per workflow/index path, not global. However, JSON.parse and JSON.stringify run synchronously on the shared serve JavaScript event loop, so this work delays unrelated workflows as well.

An API client published 738 context-capture roots in about four hours, each with a unique revision: 732 fleet contexts and six runtime contexts. The large input is intent, at 155,318 bytes. A separate fix is reducing that producer’s publication rate and replacing bulk inputs with references. This report concerns native per-save amplification for retained history. Automatic run retention already keeps one day. All affected history is younger than one day, and native run GC has no per-workflow selector.

Causal limits: one long workflow run took 957.380 seconds. Across 31 matched dispatches, 453.451 seconds elapsed before worker receipt, 159.675 seconds at the worker and 309.119 seconds afterward. Fifteen actual SSH methods took 4.628 seconds including connects. The serve Dispatching log appears before awaited worker-busy persistence and the channel call, so the 173.212-second aggregate log-to-receipt interval is not pure network or worker-queue time. These historical intervals establish the symptom. They do not establish that this index alone caused all of that latency. Historical spans do not separate every persistence, queue and initialization phase.

Source: yaml_workflow_run_repository.ts::updateIndexEntry reads the full index and calls writeRunIndex; workflow_run_index.ts::readRunIndex uses JSON.parse; writeRunIndex uses JSON.stringify of all entries. withIndexQueue keys by the resolved index path. Each WorkflowRunIndexEntry retains complete inputs.

Minimal reproduction from the pinned source checkout, using only temporary files and dummy public text:

import {writeRunIndex, readRunIndex} from './src/infrastructure/persistence/workflow_run_index.ts';
const dir = await Deno.makeTempDir();
try {
  for (const n of [1, 100, 875]) {
    const entries = Object.fromEntries(Array.from({length: n}, (_, i) => [String(i), {
      status: 'succeeded', workflowId: 'test', workflowName: 'noop', tags: {},
      inputs: {intent: 'x'.repeat(155000)},
    }]));
    const start = performance.now();
    await writeRunIndex(dir, entries);
    const loaded = await readRunIndex(dir);
    loaded!.entries['0'].status = 'succeeded';
    await writeRunIndex(dir, loaded!.entries);
    console.log({entries: n, elapsedMs: performance.now() - start});
  }
} finally {
  await Deno.remove(dir, {recursive: true});
}

Expected: updating one run should have bounded cost with respect to retained history and should not synchronously stall unrelated dispatches. An incremental or row-oriented index, or bounded indexed fields with full inputs read on demand, would avoid this amplification. No hand-edited live index, global retention reduction, frame-limit change or worker-capacity change was used.