#3874 Workflow run-start evaluation is quadratic in behavior count: provenance map construction and data-hydrating scans in spliceOrigin (~100+ s for @swamp/ci-uat/run)

Summary

Run-start evaluation of @swamp/ci-uat/run takes 102–177 s in production (median 118 s, 10 runs on 2026-10-23, swamp 20261023.090009.0-sha.9a065cc6). swamp.workflow.build_context takes ~66 ms and CEL itself takes ~50–150 ms. Almost all the remaining time is provenance bookkeeping in src/domain/expressions/splice_origin.ts. Cost grows with the square of the number of behaviors in the repo, and the work blocks the serve event loop. Typed secrets does not change it: typed and legacy runs take the same path.

Root cause

For a computed expression whose free roots include model (the UAT workflow has 7 of the form model.uat-vms.input.globalArguments.?x.orValue(...), each repeated in 4 steps), spliceOrigin does two things.

1. Quadratic map construction. bindingMapAt(context, "model", []) → modelNamespaceMap adds one withWholeData(path) per behavior field (resource, file, execution). Each call goes through graft → ProvenanceMap.of, which copies and re-canonicalizes the whole accumulated map, so building one map is O(n²). The model namespace holds every behavior twice, under its name and its UUID (model_resolver.ts, context.model[definition.id] = modelData), so n ≈ 4 × behaviors. The map is built twice per computed occurrence: 7 × 4 × 2 = 56 builds per evaluation.
2. The scan loads and walks every behavior's data. authoredRawsIn(ctx.model, …) runs valueContainsExpression / extractExpressions over the whole model root. Those read the lazy resource/file getters, so the first scan synchronously loads every behavior's records (getContentSync + JSON.parse, plus one findAllGlobalSync orphan scan). The next scans (28 per evaluation) then walk all of that data, even though every path under it is whole data and gets filtered out afterwards. A side effect: the run-start context, which later steps reuse, ends up holding run-start copies of every behavior's data, but only in workflows that happen to contain a computed model expression.

Also: WorkflowExpressionEvaluator.evaluate evaluates every occurrence (83 here) and calls spliceOrigin for each, but the workflow.expressions_evaluated span attribute reports distinct raw text (24), so telemetry undercounts the work done.

Local measurements

Probe setup: the real workflow, the real WorkflowExpressionEvaluator and typed CelEvaluator, and a context shaped like buildContext (name+UUID aliases, self-replacing loaders that parse stored JSON). These are local timings, not production CPU.

• With heavier data (20 records × 8 KB per behavior), fixing only the map still takes 33 s at 2,000 behaviors (36,000 records parsed, +576 MB heap). With the data-skipping scan it takes 1.4 s, parses nothing, and adds +97 MB.
• In every run the longest event-loop timer gap equalled the total evaluation time, so the pass never yields.
• All runs produced identical evaluated-workflow and provenance-map digests.
• Not yet established for production: the actual behavior count and data volume, so how production time splits between map building and the data scan.

Proposed fix (ranked)

1. Build the model namespace map in one pass with ProvenanceMap.fromEntries. The output is the same, because the paths are distinct two-segment paths and none nests under another.
2. Make authoredRawsIn map-aware. Skip whole-data subtrees by key before reading the value, and check leaves with scanExpressions + overlapsData. This gives the same authored-raw set, never reads a resource/file getter, and loads nothing.
3. Reuse each root's map within one spliceOrigin call (56 → 28 builds).
4. Instrumentation on swamp.workflow.evaluate: expression occurrences, actual CEL evaluations, distinct expressions, model key count, CEL ms and provenance ms. Ship this with 1–3 so production confirms the effect.
5. Follow-up, only if 4 shows it matters: scans now make up ~85% of the remaining time. Either memoize root maps and root authored raws for the duration of one evaluation pass (never keyed on the context object: a step adding model.X.execution would make a cached map read data as authored), or evaluate each distinct raw once (83 → 24) while still applying the per-path skip and foreign-text error rules to every occurrence.
6. Out of scope, needs design and security review: treating .?x.orValue(literal) on a static path as a direct read. That would flip those results from data to authored, matching the plain read, which is a visible change to masking and persisted maps.

A prototype of 1–3 with tests exists on local branch investigate-expression-provenance-perf (base 268af5537, uncommitted):

• tests in splice_origin_test.ts:
  • the new map equals the old path-by-path map, including aliases and quoted names;
  • zero getter reads during the scan;
  • zero graft calls when building the map;
• a new splice_origin_property_test.ts, which compares the new scan with the old extract-then-filter result over adversarial values and catches over-skipping mutants;
• related unit and integration suites: 1,998 passed, 0 failed.

Does typed secrets remove the need for provenance?

No. Typed secrets controls what a secret value can do in CEL and where its plaintext goes. Provenance decides whether text is author-written code.

Without provenance, ${{ vault.get(...) }} arriving in a webhook payload or a stored resource would be evaluated by a later pass under the run's authority and loaded into a method argument, a trusted destination. Typed secrets also doesn't cover env.*, cross-behavior data.* reads, file.contents, or foreign text such as ${{ github.sha }} that must stay literal.

All of these still have to hold:

• the raw-text rule;
• position ranges, because data can equal an authored raw;
• the partial-overlap rule for fragments;
• parent maps carried into child workflows;
• persisted maps for resume.

Section 8.1 of the typed-secrets design adds provenance work rather than removing it. Once the untyped branch is deleted (typed-secrets 4.6), the separate raw and sanitized copies, each with its own map, could merge into one. That is a representation simplification and doesn't affect this perf issue.

Acceptance

• Evaluation of the UAT workflow stays roughly linear in behavior count, with no resource/file loads during the provenance scan.
• Provenance output is unchanged: the equivalence and property tests pass, and the provenance/cache/resume integration suites stay green.
• swamp.workflow.evaluate reports occurrences, evaluations, model keys and subphase timings.
• The readiness-step pre-dispatch gaps (110–161 s) look like a separate problem: no correlation with evaluation time (r = 0.02) and no overlap with other runs' evaluations. Track them separately.