#3617 v3 local durability: chunks aren't fsynced before the commit naming them, so a power cut can wedge every checkout; torn writer and format markers don't heal (bug hunt group 4)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). A v3 datastore stores content as chunks (_blobs/ on a filesystem, the provider's storage() on S3/GCS) and commits that name them in commits.db, which is fsynced (synchronous=FULL). Each checkout rebuilds its files from the commits (the read model). Several small local files steer recovery:

• the format marker (datastore-format.json);
• the writer marker (writer.json, swamp-club#3299);
• the read model's unsynced list and reboot check.

A v3 bug hunt (2026-10-16, verified on main d3b76490c; the code below is unchanged on current main) found that these files aren't written durably, and their failure modes don't heal. They're live on filesystem v3 today. On S3 and GCS, chunks are single object PUTs and so durable on acknowledgement, but commits.db's local copy, the outbox, the markers and the read model are still local files.

The bugs

1. Chunks are never fsynced before the commit that names them (V3-01, S1). FileSystemStorage.put (fs_datastore_storage.ts) writes through atomic_write.ts (write then rename, no file or directory fsync), while the commit naming the chunk is FULL-synced. A power cut in the writeback window leaves the commit durable and the chunk 0 bytes or missing.

• Reproduced (one verifier end to end with the compiled binary):
  • every v3 context open fails at catch-up with a ReadModelApplyError ("holds 0 bytes"), so data list, model list, datastore status, datastore gc and model method run all exit 1, in the writer's checkout and in fresh clones;
  • doctor datastores reports healthy;
  • deleting the read model replays into the same error;
  • rewriting the content only touches the 0-byte chunk, which it treats as present.
• Detection checks existence only (fs_datastore_storage.ts has, chunk_store.ts missing/#store, the outbox recovery's chunk checks). So recoverCommitOutbox lands a prepared commit that names a 0-byte chunk, and GC never reads referenced chunks: v3_datastore.ts passes no verifyBytes, so collectChunks reports damaged: 0.
• The wedge: sqlite_read_model.ts wraps any apply error as retryable, and v3_repo_context.ts awaits catch-up on every open, so one unappliable commit blocks every command.
• Trigger: power loss or kernel panic within ~5–30 s of any v3 write storing over 8 KiB, on default ext4/XFS. Bit rot or a partial copy wedges the same way.
• swamp-club#3586's leftover belongs here: outbox recovery only warns about a landed commit naming a missing chunk.

2. The writer marker isn't fsynced, and a torn one reads as "no marker" (V3-09, S2). writeWriterMarker (interrupted_writer_recovery.ts) uses atomicWriteTextFile with no fsync. A 0-byte writer.json makes readWriterMarker return undefined ("Ignoring the unreadable v3 writer marker"), recovery returns early, and recoverAndMark then overwrites it. Reproduced: with a 0-byte marker, a rewritten checkout file still reads as uncommitted and a new uncommitted file stays, for good. The checkouts never converge.

3. The format marker isn't fsynced (V3-10, S3). datastore_format_marker_writer.ts writes then links with no file or directory sync on the main path; the fallback syncs the file but not the directory. Reproduced: with a 0-byte datastore-format.json, every command in every checkout is refused ("not a valid format marker"), datastore setup refuses to move the repo, and doctor datastores fails. Only writing the JSON back by hand fixes it. A missing marker gives datastore_format_mismatch, whose advice (re-run setup) is then refused.

4. A clean reboot triggers a false reset and full replay (V3-14, S3). A writer applies its own commits to its read model only at the next process's catch-up. The unsynced list is cleared only by serve's timer, and the reboot check runs before catch-up. So a writer's legitimate overwrite of a key an earlier catch-up listed mismatches after an ordinary reboot. The read model resets and replays everything, with a warning that claims crash damage. Reproduced. The trigger is ordinary use: the last command before shutdown wrote a key (a latest marker, a definition, a run record) that an earlier catch-up listed.

5. A removal a power cut undid is never redone after compaction (V3-36, S3). After a reset, state.reset empties ownership, #loadBase takes removals from loadRemovals, which is now empty, and #sweepUnowned runs only in recover(). Reproduced: with compactHistory, a deleted record's file survives the reset and base load for good; without compaction it's removed. The same gap affects the unnamed-boot and generation resets.

What to do

1. Chunks durable before they're named (1). fsync each chunk's temp file and its shard directory before the unit's outbox.prepare / append: one durable barrier per unit, not per chunk where avoidable. The read model's batched fsync (swamp-club#3337) is the pattern. Measure the cost against the uat#496 baseline (the v3 write and first-clone lines, macOS and Linux) and give before and after in the PR.
2. Torn chunks count as missing (1): has(), missing() and #store compare the stored size with the BlobRef (and the hash where it's cheap), so a rewrite re-puts a torn chunk and outbox recovery drops a prepared entry naming one as missing-chunk. For a landed commit with a missing or torn chunk, recovery stores it again from the checkout's file when that file still has the content, and otherwise warns, as today.
3. One bad commit must not wedge every command (1): the read model treats a permanently missing or damaged chunk on a landed commit as a per-key failure: that record unreadable, with an error naming it and the way out. It must not block the commits after it. Say how a user repairs it, and make doctor datastores detect it.
4. GC verifies referenced chunks (1): pass a rotating verifyBytes budget from the CLI's closing pass and serve's maintenance timer, so damage is found over time.
5. Writer marker (2): fsync writer.json and its directory once per command. Treat an existing but unparseable marker as interrupted and recover conservatively, and never overwrite it before recovering.
6. Format marker (3): sync the temp file before the link and the directory after it. Let datastore setup or doctor datastores --repair rewrite an empty or missing marker beside a valid commits.db, and correct the mismatch error's advice.
7. Clean reboot (4): when the writer commits a key, update or forget its unsynced-list entry, or have the reboot check accept the bytes of a committed-but-unapplied put of that key.
8. Removals after a reset (5): after a reset in a compacted datastore, sweep unowned admitted files during base load, or keep prior ownership rows as removal candidates. Cover the unnamed-boot and generation resets too.
9. Compiled binary: any new /proc or /dev access goes through the existing FFI path (the compiled binary's flags refuse direct reads).

Tests

Each must fail on main first. Power cuts are simulated by planting the torn state (0-byte or missing files), as the hunt did. Put child-process cases in integration/.

• 1:
  • a commit naming a 0-byte chunk: catch-up doesn't wedge, the other records read, and the damaged record names its problem;
  • a rewrite re-puts the torn chunk;
  • outbox recovery drops a prepared entry naming a torn chunk;
  • recovery restores a landed commit's missing chunk from the checkout file;
  • GC with a verifyBytes budget reports the damage;
  • a fsync spy shows the chunk and directory synced before prepare, with one barrier per unit.
• 2: a 0-byte writer marker is recovered as interrupted, not ignored or overwritten.
• 3: a 0-byte or missing format marker beside a valid commits.db is repaired by setup or doctor, and the error advice matches.
• 4: a clean "reboot" (boot identity changed, files intact) after a writer overwrote a listed key causes no reset.
• 5: after a reset in a compacted datastore, a file whose removal was undone is removed.
• swamp-uat passes unchanged.

Severity: S1 (a power cut can wedge every checkout, with no supported recovery), one S2, and three S3s. This must land before v3 is announced (datastore rework Phase 4 I).

Blocked by: nothing. Bug hunt groups 7 (namespaces) and 8 (recovery-sweep scope), not filed yet, also change sqlite_read_model.ts and follow this one.