#3357 Datastore rework Phase 4: measure Litestream replication on real S3 and GCS buckets (spike: per-command vs idle sidecar, cost of waiting for replication)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits.

Phases 1–3 are done (main at fc312a85). v3 works today for new, empty filesystem datastores with a single writer (swamp-club#3279, and the fixes #3299, #3300, #3301, #3337). It's tested in CI by swamp-uat's filesystem-v3 backend.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream continuously replicates it to the S3 or GCS bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease in the bucket's control plane makes sure only one machine writes.

The decisions so far are in design/enablers/datastore-litestream.md (swamp-club#3234) and in design/enablers/datastores.md, under the "v3 …" sections.

Phase 4 split (all on #2865's plan):

• A: real-bucket latency spike.
• B: Litestream binary management.
• C: fenced writer lease.
• D: Litestream sidecar and replication (needs A, B and C).
• E: v3 on S3 and GCS (needs D, plus the extensions' storage/1 and swamp-club#3204).
• F: durability leftovers.
• G: extension work.
• H: swamp-uat s3-v3 and gcs-v3.
• I: dogfood and benchmark.

A, B and C run in parallel.

A decision for Phase 4: a v3 write command waits for Litestream to confirm replication (POST /sync {"wait": true}) before it reports success. One sync covers every commit the command made. The design promises that no acknowledged write is lost, and a dead laptop must not take acknowledged writes with it. Read-only commands never wait.

Goal

Measure what Phase 4 costs on real S3 and GCS buckets, so D can choose how swamp runs Litestream:

• per command: start the sidecar, force one sync, stop;
• or idle: a sidecar that stays up for a short idle period behind a pidfile, so commands reuse a warm connection.

This is a spike. It produces measurements and a decision, not production code. Prototypes stay on a branch that isn't merged.

What the Litestream spike already measured (emulators only)

These numbers are in datastore-litestream.md:

• Forced sync: 9–15 ms on local emulators. About 44 ms with a warm connection at 30 ms RTT, and about 165 ms on a cold per-command connection.
• Whole per-command sidecar: 17–30 ms on Linux and macOS arm64, about 197 ms with 30 ms RTT, and 110–150 ms on Windows.
• Real buckets add TLS setup and credential resolution (SSO refresh, IMDS, workload identity) to the first request. This spike measures exactly that.

Questions to answer, with numbers (median and p95, at least 20 runs each)

1. Forced sync on a real bucket, one commit and a burst of commits:
   • S3 (two regions: one close, one far) and GCS;
   • per-command (cold) against idle-sidecar (warm);
   • with the credential chains users really have (static keys, SSO profile, instance or workload identity).
2. A whole v3 write command, using a local filesystem-v3 prototype, with Litestream replicating to the real bucket: model method run writing one small and one large output, and data delete.
   • Compare against today's v2 S3 and GCS push for the same command, from swamp-uat#496's baseline numbers or a re-run against the same buckets. The wait replaces v2's push, so that's the comparison that matters.
3. Restore time from a real bucket: 1 MB, 100 MB and 1 GB, plus the follow-mode lag (restore -f). Phase 4's takeover uses this.
4. Replica growth, with SQLite's default checkpointing, over a realistic day of commits.
5. The idle-sidecar design, if it wins:
   • the idle timeout;
   • how a command finds and reuses the sidecar (pidfile plus socket);
   • how it detects a dead one;
   • what happens on macOS sleep and wake.

Deliverable

Add a section to design/enablers/datastore-litestream.md, linked from #2865, that records:

• the numbers;
• the choice for D, per-command or idle sidecar, and why;
• the cost of waiting for replication compared with v2's push, saying plainly whether it's acceptable;
• anything that changes D's or E's plan.

Put the branch name and the measuring commands in the record, so the numbers can be reproduced.

Notes

• You need real S3 and GCS buckets and credentials for a test account. Never use production data.
• Emulator numbers don't count for this issue.
• Use the Litestream release the binary-management issue (B) pins: 0.5.17, or 0.5.18 if it's out.

Blocked by: nothing. It runs in parallel with B and C.