#3935 v3 on S3/GCS: a replica restore yields a corrupt commits.db (fails SQLite's integrity check), so fresh clones and restores of the datastore fail; reproduces with Litestream alone from its first compacted L1 file (found by #3916)

What happens

A fresh clone or restore of a v3 bucket datastore fails, on S3 and on GCS:

Restoring the v3 datastore copy failed: the restored database failed SQLite's integrity check: Tree 2 page 17040 cell 7: 2nd reference to page 4131 / Rowid 4487 out of order …

On GCS: "Page 21480: never used …".

• The writer's own commits.db is intact (quick_check passes, about 6,400 commits), so nothing is lost.
• But nobody else can clone the datastore, or restore their checkout from it, until a new lineage replaces the broken one. For anyone but the writer, the datastore is unavailable.

swamp's restore is behaving correctly: its integrity check refuses the corrupt copy instead of installing it.

Found while measuring swamp-club#3916 (on the dogfood repos that swamp-club#3756 populated). It isn't caused by #3916: the branch never wrote to these buckets, and main (4f7649ed1) and release 20261024.011325.0 fail identically.

It reproduces with Litestream alone

On the S3 replica (dogfood-s33/s33-572700, lineage 000000001570, head txid 0x70), restoring with Litestream 0.5.17 alone, outside swamp:

The lineage holds:

• L9: 0001-0002
• L2: 0001-0071
• L1: 0001-001a, 001b-003f, 0040-005f, 0060-0070, 0071-0071
• L0: 0071

So the snapshot is sound, and the damage arrives with the first compacted L1 file. Either Litestream's compaction wrote it corrupt, or its restore across compacted levels applies it wrong.

This fits a note open since swamp-club#3234: Litestream's snapshot compactor logged "compaction failed … read page … EOF" under forced checkpoints, and once in a failover run.

It looks intermittent across the history. During population with one writer:

• restores succeeded at about 2,400 and 5,600 commits;
• they failed at about 4,400, 5,300 and every attempt at about 6,400.

Repro state (kept)

• Containers: df3916-s3 (ministack, :47511) and df3916-gcs (fake-gcs 1.56.1, :47512, volume df3916-gcs).

• Writer checkouts and failing probes: under /private/tmp/claude-501/-Users-stack72--herdr-worktrees-swamp-3916/8b0f9145-eec5-43dc-906e-54bcc40d9a79/scratchpad/df/:

  • repos/s33 and repos/gcs3;
  • probe-s33, probe-gcs3 and probe-rel-s33.

  Scratchpads get wiped. Copy what's needed somewhere durable first, such as ~/dogfood-3756/.

• Litestream: 0.5.17, as swamp pins it.

Work

1. Find where it breaks:
   • which L1 file's pages differ from the writer's commits.db at that txid (decode the LTX pages and compare them);
   • so, whether the file was written corrupt (compaction) or is restored wrong (level selection, or overlap with the 0001-0002 snapshot).
2. Check Litestream upstream for a fix after 0.5.17 (compaction and restore across levels).
3. Reproduce it from scratch, in a test or a scripted repro:
   • populate a v3 bucket datastore with forced checkpoints and swamp's own history compaction;
   • restore at each level boundary;
   • include a run with swamp-club#3916's policy (500/500), which folds swamp history far more often and frees many pages. If page churn triggers this, #3916 makes it more frequent.
4. Fix it:
   • upgrade or patch Litestream if it's upstream;
   • change how swamp drives Litestream if it's the configuration or checkpoint pattern.
5. Detect it before readers do, not only at their restore. A writer that can't produce a restorable replica should notice; for example, a periodic restore-verify of the current lineage on the writer, or after compaction. When the replica is found broken, recover by starting a new lineage from the writer's intact copy. Report it through doctor datastores.

Done when

• The cause is identified, and fixed or worked around.
• A scripted repro (forced checkpoints plus compaction) passes many restore-verify rounds on S3 and GCS, with today's and #3916's compaction policy.
• A broken replica is detected on the writer and recovered with a new lineage, with a test.
• The preserved buckets either restore after the fix, or are recovered by the new detection.

Severity: high. Nothing is lost, but every other checkout of the datastore is locked out. It blocks everything in team mode (Phase 5) that runs on buckets: the team-mode dogfood with its real-bucket pass, and the announcement.

Blocked by: nothing.