#3358 Datastore rework Phase 4: download, verify and cache a pinned Litestream binary on first v3 use (dormant)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits.

Phases 1–3 are done (main at fc312a85). v3 works today for new, empty filesystem datastores with a single writer (swamp-club#3279, and the fixes #3299, #3300, #3301, #3337). It's tested in CI by swamp-uat's filesystem-v3 backend.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream continuously replicates it to the S3 or GCS bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease in the bucket's control plane makes sure only one machine writes.

The decisions so far are in design/enablers/datastore-litestream.md (swamp-club#3234) and in design/enablers/datastores.md, under the "v3 …" sections.

Phase 4 split (all on #2865's plan):

• A: real-bucket latency spike.
• B: Litestream binary management.
• C: fenced writer lease.
• D: Litestream sidecar and replication (needs A, B and C).
• E: v3 on S3 and GCS (needs D, plus the extensions' storage/1 and swamp-club#3204).
• F: durability leftovers.
• G: extension work.
• H: swamp-uat s3-v3 and gcs-v3.
• I: dogfood and benchmark.

A, B and C run in parallel.

A decision for Phase 4: a v3 write command waits for Litestream to confirm replication (POST /sync {"wait": true}) before it reports success. One sync covers every commit the command made. The design promises that no acknowledged write is lost, and a dead laptop must not take acknowledged writes with it. Read-only commands never wait.

Goal

swamp can obtain a verified Litestream binary when, and only when, a v3 datastore needs replication. It stays dormant until the sidecar issue (D) uses it.

The decision this implements (#2865, 2026-10-11)

This replaces #3234's "embed with deno compile --include" choice. Embedding would add about 38 MB to every release for every user; downloading means only v3 users pay. Fetching upstream's Apache-2.0 release also avoids redistributing it. Embedding is revisited at Phase 7, when v3 becomes the default.

1. Pinned at build time. The swamp release records a Litestream version and the SHA-256 of each platform's release archive and binary, for darwin-arm64, darwin-x86_64, linux-arm64, linux-x86_64 and windows-x86_64. Keep it in one source file, and add a build check (or script) that refreshes and verifies it from upstream's checksums.txt.
2. Downloaded on first use, from upstream's GitHub release for that version and platform:
   • verify the SHA-256 on download;
   • extract into a per-version directory under ~/.swamp/ (for example ~/.swamp/litestream/<version>/), writing atomically: temp, then rename;
   • verify the binary's SHA-256 again before every start. It sees bucket credentials and every commit, so a swapped binary must never run.
3. Offline and air-gapped machines:
   • an override path, through an environment variable or config, points at a binary the user downloaded themselves;
   • it's still hash-checked against the pinned value;
   • with no network and no override, the error says exactly what to download, from where, and where to put it.
4. Concurrency. Two processes downloading at once don't corrupt the cache. Use a lock file, or let both download and have the rename settle it.
5. The serve Docker image bakes in the pinned binary and points the override at it, so serve never downloads at runtime. Update the Dockerfile.
6. Windows. Litestream before 0.5.18 can't replicate on Windows (litestream#1542). If the pinned version is below 0.5.18, Windows reports that v3 replication isn't available yet, and stays on v2. Record the platform rule.
7. Docs. Update design/enablers/datastore-litestream.md: replace the "embed" decision with this one, and mark the earlier "superseded for shipping" note resolved.

Tests

• Unit tests, with an injected fetch and filesystem:
  • download, verify and extract;
  • a hash mismatch on download, and on a cached binary, is refused and never run;
  • the override path is honoured and hash-checked;
  • the offline error;
  • concurrent first use;
  • per-version directories coexist.
• Opt-in network check: a test, skipped by default, that downloads the pinned release and checks the pinned hashes match upstream.
• Dormancy: nothing in production calls it yet. Extend integration/datastore_v3_rules_test.ts.
• v2 doesn't change.

Out of scope

Running the sidecar, replication, restore, and the writer lease (issues C and D).

Blocked by: nothing. It runs in parallel with A and C.