#3417 Datastore rework Phase 4: real-endpoint check of Phase 4A's replication figures on a hosted S3-compatible service (spike)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits. v3 is opt-in today for filesystem datastores (swamp-club#3279), and swamp-uat's filesystem-v3 backend tests it in CI.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream replicates it to the bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease makes sure only one machine writes.

Done so far:

• A, swamp-club#3357: replication cost on a bucket model, choosing an idle sidecar. See design/enablers/datastore-litestream.md, section "Phase 4A".
• B, swamp-club#3358: download, verify and cache a pinned Litestream (0.5.17), dormant: src/infrastructure/datastore_v3/litestream/litestream_binary_provider.ts.
• C, swamp-club#3359: the fenced writer lease (src/domain/datastore_v3/writer_lease.ts, src/infrastructure/datastore_v3/v3_writer_lock.ts). Every CLI write command takes a new fence, and serve keeps one for its lifetime. commits.db keeps a store-wide writer_fence.

The decisions are in design/enablers/datastore-litestream.md, under "Decision" and "What changes for Phase 4D and 4E", and in design/enablers/datastores.md, under the "v3 …" sections.

Goal

Confirm Phase 4A's modelled figures on real endpoints before Phase 4E puts v3 on S3 and GCS. Phase 4A (swamp-club#3357) measured an emulator behind a latency, handshake, bandwidth and credential model. Its request, byte and connection counts carry over to real buckets; its times are modelled.

This is a spike. It produces measurements and an updated decision record, not production code. Prototypes stay on a branch that isn't merged.

The S3 side runs on a hosted S3-compatible service, not AWS. S3-compatible storage is one of swamp's promises, so these numbers matter on their own:

• use a hosted service (for example Cloudflare R2, Wasabi, Backblaze B2 or Tigris), with one region near the machine and one far from it;
• Litestream and swamp's S3 extension both take a custom endpoint;
• never use production data.

Reuse Phase 4A's harness

The scripts are on branch spike/3357-litestream-buckets (never merged; spike/README.md lists them, with the commands in the record's "Reproducing" section). Point them at the real endpoint instead of the proxy.

Use:

• the Litestream release swamp pins (0.5.17, or 0.5.18 if it's out and pinned by then);
• the fixed teardown order: sync, close the stores, then stop or release the sidecar.

What to measure (median and p95, at least 20 runs each)

1. Forced sync, cold and warm, near and far, on the S3-compatible service with static keys:

   • cold: start a sidecar, write, sync, stop;
   • warm: sync on a running sidecar.

2. Idle connection lifetime: how long the endpoint keeps a pooled connection. Measure the warm sync after idle gaps of 1–300 s. This sets how much of the warm figure an idle sidecar keeps.

3. A whole v3 write command (the 4A "Q2" rows: small, delete, 10 MB), with an idle sidecar and with one per command. Compare with v2's push to the same endpoint through @swamp/s3-datastore with its custom endpoint.

4. Restore: 1 MB, 100 MB and 1 GB from the far region, plus follow lag at the default 1 s poll.

5. GCS, if a test GCP project is available:

   • cold and warm sync with application default credentials, including what the ADC token fetch adds;
   • uploads over 16 MB (the emulator bypassed the model there);
   • restore at 100 MB.

   If there isn't one, say so, and leave GCS listed as an open risk for 4E.

6. The reversal figures in "When the choice reverses": a cold forced sync under 250 ms at p95 near. Report whether the idle-sidecar choice stands.

Not covered, by decision

• AWS's own credential chains (SSO refresh, instance or workload identity) stay modelled. An idle sidecar resolves credentials once per session, not per command, so they barely move the choice. Say so in the record.
• AWS's own endpoint behaviour (idle timeout, TLS) may differ from the S3-compatible service's. Say which numbers that affects.

Deliverable

Update design/enablers/datastore-litestream.md:

• real figures beside the modelled ones, in the "Real-bucket check" section;
• the idle-sidecar decision confirmed or reversed, with the reason;
• anything that changes D's or E's plan;
• the branch, the service and region used (no credentials), and the commands to reproduce.

Blocked by: nothing; needs a test account on a hosted S3-compatible service (and a GCP test project for the GCS part). Runs in parallel with D1.