#3467 Run index snapshot cache keeps serving a stale run status: ctime is only second-resolution so in-place same-size rewrites alias the stamp, and snapshots are cached even when repair could not write the index (regression from #3105)

Summary

Since #3105 (c5038e5b, swamp-club#3443), the run index snapshot cache in swamp serve / the swamp dashboard child (RunIndexCache, src/infrastructure/persistence/run_index_cache.ts) can keep serving a stale run status after the run's record and index have changed on disk. Search, summary and the dashboard show the old status until something else moves the cache stamp (index stat or records-dir mtime). Before #3105 every read re-derived from disk and was correct. Two independent ways in:

1. Stamp aliasing: Deno reports ctime in whole seconds, and the racy check looks only at mtime

RecordFingerprint compares {mtimeMs, size, ctimeMs, ino}. Deno 2.9.7 on Linux reports ctime truncated to whole seconds (the kernel shows …:32.802, Deno gives …332000), while #racy() only distrusts a recent mtime (or dir mtime). So an in-place, same-size rewrite that restores the old mtime (cp -p / rsync --inplace -t style), landing in the same wall-clock second as an earlier metadata change after which the snapshot was cached, leaves the stamp identical forever.

Repro (serve on a 2-workflow × 20-run repo, cache warm, files settled > 2 s):

1. Align to the start of a second. Rewrite the workflow's .runs-index.json in place with identical content and restore its mtime (Deno.utimeSync). Its ctime moves to this second; its mtime is old, so it is not considered racy.
2. Request workflow.run.search {query: <runId>} → loads and caches the snapshot (status succeeded).
3. Within the same second: rewrite that run's record and the index in place, flipping status: succeeded → status: cancelled (same byte length), and restore both mtimes.

Result: serve returns succeeded immediately, at +3 s and at +10 s (3 of 4 attempts persistent; the 4th missed the second). Records and index both say cancelled. Stat before/after step 3 is identical: {"mtimeMs":1791999229925,"ctimeMs":1791999293000,"size":6331,"ino":157728497}. OLD (3999f85e) returns cancelled. On Windows, where the code expects no ino/ctime, any same-size, mtime-restoring rewrite would alias, without needing the same-second timing.

The same ctime truncation affects record fingerprints used by repair-by-stat (findSuspectRecords).

Fix ideas: treat a recent ctime (or a ctime with no sub-second part, within 1 s of now) as racy too; or don't rely on ctime below one-second resolution.

2. A snapshot is cached even when repair could not bring the index in line

When getValidatedIndex → repairIndex can't persist its repair (e.g. read-only index/runs dir: "Failed to write repaired index, will retry next read"), the in-memory repaired index is still turned into a snapshot and cached under a stamp that won't change (the index file isn't rewritten, and the dir mtime doesn't move). Later in-place edits to records in that dir are then never seen.

Repro (serve, 2 × 20 repo, cache warm):

1. Add one run record by hand with no index entry, then chmod 555 the runs dir.
2. Read twice > 2 s apart, so the repaired-in-memory snapshot is cached.
3. Edit another run's record in place (status: succeeded → status: failed; the file stays writable, the dir doesn't).

Result: serve keeps returning succeeded for the whole 70 s observed. OLD returns failed on every read (it re-derives on each count mismatch).

A related variant: a 0-byte workflow-run-<id>.yaml (a crashed writer, or a non-atomic sync that creates the file first) is cached as-is. When it is later filled in place, the run stays invisible for about 25 s, until an unrelated periodic serve task rewrites the index. OLD shows it immediately.

Fix idea: don't cache a snapshot when the record count still disagrees with the index after repair, or when the repair's write failed.

Environment

Linux x86_64 (ext4 on NVMe and tmpfs), Deno 2.9.7, swamp at 37627d76 ≡ c5038e5b. Found during adversarial e2e verification of #3105; OLD = 3999f85e.