#3337 Datastore rework: a fresh v3 checkout is slow because catch-up fsyncs every file one at a time (27 s vs 1.1 s for 1,000 items on macOS)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). The new format, v3, records every change as a commit in a single-writer SQLite commits.db, with synchronous=FULL (src/infrastructure/datastore_v3/sqlite_commit_store.ts). Each checkout rebuilds its local files under .swamp/, and _catalog_v3.db, from those commits through a read model (src/infrastructure/datastore_v3/sqlite_read_model.ts, swamp-club#3239). It catches up whenever a context opens.

The read model's state, including its cursor and the live-record index with each present record's sha256, lives in .swamp/_read_model_v3/_read_model_v3.db. v3 is released for new, empty filesystem datastores (swamp-club#3279; main at 6ada06a5).

How files are written today. Every file the read model materialises goes through writeFileDurably (src/infrastructure/datastore_v3/atomic_stream_write.ts), one at a time, from the per-commit loop in sqlite_read_model.ts (around #writeContent, ~line 680). Per file it:

1. writes a temp file beside the target;
2. calls file.sync();
3. renames the temp file into place;
4. calls syncDirectory(dir);
5. calls syncDirectory(parent) for every directory it created.

The guarantee this buys, from the module doc: the read model advances its cursor only once a commit's files are durable.

The problem (found by the swamp-uat#496 baseline, PR swamp-uat#640)

The benchmark's "first clone" case is a second checkout of a datastore that holds 1,000 data items in one commit. It times setup plus the clone's first swamp data list, which does the catch-up.

The catch-up writes 3,015 files in 2,020 directories: latest, 1/metadata.yaml and 1/raw per item.

• On macOS, FsFile.sync() is F_FULLFSYNC, which flushes the drive's cache. A microbenchmark on the same Mac measured 7.52 ms per durable file against 0.08 ms for a plain write. 3,015 × about 7.5 ms is about 23 s, and the parent-directory syncs for about 2,000 new directories bring it to the 27 s measured.
• On Linux, fsync is cheaper, which is why the gap is about 3×.
• Steady state (a model method run, peer and delete visibility) is close between v2 and v3. The gap is specific to replaying many records.

This applies to every catch-up after other writers' commits, not only a fresh clone. Phase 4 puts many clients on one datastore, which makes catch-up a hot path. In Phase 3 a writing command also re-applies its own commit through this path, so check that cost too.

Goal

Under the Phase 4 entry criterion: a fresh filesystem-v3 checkout of 1,000 items within about 2× of v2 on both macOS and Linux, measured by the swamp-uat#496 benchmark line "first clone: setup + data list (1000 items)", without weakening crash safety.

The invariant that must hold

The read model never records a commit as applied while that commit's files could still be lost to a crash, OS crash or power loss included. If it did, a checkout could believe it's up to date with a missing or empty file, and never repair it.

That's why simply dropping the fsyncs isn't acceptable. _read_model_v3.db is itself durable (SQLite FULL), so its cursor could survive a power loss that the unsynced files didn't. A "clean shutdown" marker doesn't help either: a process can exit cleanly while its data is still in the OS page cache.

Design

Measure each step with the benchmark before going further.

1. Batch the syncs. For each catch-up page (or each commit, whichever is larger):

   1. write every file with no sync: temp file, then rename;
   2. fsync the written files, with bounded parallelism;
   3. fsync each touched directory once, deepest first, including newly created parents;
   4. only then advance the cursor and the live-record index.

   The invariant is unchanged. Keep atomic temp-plus-rename for each file, so a reader never sees a torn file.

2. Measure on macOS. If macOS is still well over 2× after step 1, because F_FULLFSYNC calls don't coalesce, add a durable checkpoint and leave the per-file syncs out of the hot path:

   • keep a separate "durable up to" cursor in the state database, advanced only at a sync barrier, for example at the end of a catch-up;
   • when a checkout opens, re-apply the commits after the durable cursor (writes are idempotent; replaying a commit twice changes nothing, #3239), or check the files written since the barrier against the live-record index's sha256;
   • prove that a crash at any point (before the barrier, during it, after it) leaves a checkout that repairs itself on its next open.

   Record the choice and why in design/enablers/datastores.md ("v3 read model").

3. Don't touch commits.db durability. It stays synchronous=FULL and remains the source of truth.

4. Windows. syncDirectory already treats a directory sync as best effort where it can't be done. Keep it working there; #495 runs the suite on Windows.

5. v2 doesn't change. This is v3 read-model code only. If anything shared changes, the v2 gate from #3240 and #3250 applies.

Tests

• Every existing read-model crash-safety test passes unchanged: an error after a file write and before the cursor moves, followed by a catch-up, gives the same result as a clean run. Add tests for the new barrier: a failure between the writes and the barrier, and during the barrier, leaves a state that repairs itself on the next open.
• If you add the durable checkpoint, simulate a lost write, for example by truncating a file written after the last barrier, and show that the next open repairs it.
• The property test against the reference model still passes.
• Benchmark: report the swamp-uat#496 "first clone" numbers, macOS and Linux, v2 against v3, before and after, in the PR. Also report the steady-state model method run line, to show it didn't regress.
  • Locally, from the swamp-uat repo: SWAMP_SOURCE_DIR=<swamp checkout> SWAMP_TEST_BACKENDS=filesystem,filesystem-v3 SWAMP_BENCH_SAMPLES=3 deno task uat:benchmark:remote.
  • In CI: the Remote Datastore Benchmarks (filesystem-v3) job in swamp-uat's benchmark.yml, including its daily 10k-item run.

Out of scope

• Phase 4's embedded service, the outbox and Litestream.
• commits.db's own durability.
• F_BARRIERFSYNC, which Deno doesn't expose; it would need FFI.

Done when

• The first-clone benchmark is within about 2× of v2 on macOS and Linux, with numbers in the PR.
• The invariant holds, proven by the crash tests above.
• The design is recorded in datastores.md.
• All existing suites pass unchanged.

Blocked by: nothing. Do it before Phase 4's service work starts.