#3443 Dashboard Workflows, workflow and run pages are slow with large history: workflow.run.summary and workflow.run.search read every run index of every workflow on each request

Description

The dashboard's Workflows page, a workflow's detail page and a run's page are slow on repos with many workflows or a long run history. The cause is on the server. workflow.run.summary and workflow.run.search read every run of every workflow, for all time, on every request. Only then do they apply the time window, the workflows/workflow filter and the limit. So a request costs the same whether it asks for the last 24 hours of one workflow or for everything.

This is related to #3407 but is not the same issue. #3407 is about run records being YAML-parsed. Here, serve already uses the run index (findAllSummariesFromIndex, since #1310), and the index path still grows with total history.

Where the time goes (per request)

1. Every run index is read in full. workflowRunSearch (src/libswamp/workflows/run_search.ts) calls findAllWorkflows(), then findAllRunsByWorkflowId for every workflow through collectBounded. For each workflow, findAllSummariesFromIndex → getValidatedIndex does a readDir of the runs directory to count records, then reads and JSON.parses the whole .runs-index.json. indexToSummaries then builds Dates for every entry and sorts them. All of this happens before any filter runs.
2. Every run becomes a search item before filtering. Each run gets copies of its tags and inputs, ISO strings and nested-wait resolution. Then since, until, workflow and workflows filter the list in JS. limit slices last.
3. One-workflow requests pay the whole cost. workflows: [name] and workflow: name are only post-filters. The loader is never limited to that workflow's id.
4. Every workflow file is re-read and re-parsed on each request. workflowRepo.findAll() → YamlWorkflowRepository.readAll() has no cache. workflow.search, workflow.run.search and workflow.run.summary each call it, the last two only to build workflowsWithInputs.
5. With auth on, renamed or deleted workflows force a full scan. filterByResources → CanonicalResources.workflowOwners → resolveRecordedWorkflow → findByName. When the fast path misses (runs recorded under a workflow's old name after a rename, or a name that is not filename-safe), this falls to findAll(), a full scan of the workflows directory. That happens once per distinct (id, name) in the result. The result is memoized per request only.
6. Serve runs this on its only thread. Other requests queue behind it, as #3407 also saw.

How the dashboard multiplies it

• Workflows page: workflow.run.summary { since, until, recentLimit: 20 }. useTimeBounds rounds to the minute, so on the first render after each minute rolls over (and health-stream renders happen constantly) the request key changes and it refetches. That is one full scan per minute per open tab.
• Workflow detail: workflow.run.search { workflow, limit: 20 }. This is a full scan for 20 rows.
• Run detail: workflow.run.search { workflow, query: runId, limit: 1 }, plus the recent-runs strip's workflow.run.summary { workflows: [name], allTime: true, recentLimit: 30 }. While the run is live, the run search is polled every 2.5 s (LIVE_POLL_MS). That is a full scan of all history every 2.5 s, to read one row.

Measured

I timed the real workflowRunSummary / workflowRunSearch with YamlWorkflowRunRepository.findAllSummariesFromIndex against generated .runs-index.json files. Workflow loading and auth were stubbed out, so these numbers are a lower bound. Local NVMe, Linux, warm cache, current main (no relevant changes since 894ca3b0).

The cost is linear in total runs kept, about 4 µs per run. It does not change with the window, with the number of workflows asked for, or with the limit. Real index entries carry larger inputs, failureReason and so on, and slower disks or a remote datastore add more. Workflow YAML parsing and auth come on top of these numbers.

Steps to reproduce

1. Use a repo with about 100k runs across many workflows. Any split works.
2. Open swamp serve / swamp dashboard → Workflows. Then open one workflow, then a live run.
3. Watch swamp.workflow.run_summary / swamp.workflow.run_search spans. Every request costs the same, a live run page repeats it every 2.5 s, and the Workflows page repeats it every minute.

Expected

• Push the filters down to the loader:
  • workflows/workflow should load only those workflows' indexes.
  • since/until should skip entries before converting them, or, better, skip whole workflows whose index says their newest run is older than since.
  • The limit should apply while merging newest-first, not after building every item.
• A run lookup by full id (query: runId, limit: 1) should not scan all runs. findGlobalById already does this lookup with a single stat per workflow.
• Keep parsed run indexes and workflow definitions in memory in serve, invalidated by the existing write and markDirty hooks, so repeated dashboard requests don't re-read and re-parse unchanged files.
• Request cost should grow with the result size, not with total history.

Related: #3407 (run search YAML parsing), #1310 (index-backed run search), #3051, #3408.