#3756 Datastore rework Phase 4 I: dogfood and benchmark solo v3 (filesystem, S3, GCS) in a throwaway repo with the released binary, check the kill criteria, and decide whether to announce

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished its solo-mode code (Phase 4):

• v3 on filesystem and on S3/GCS (#3534);
• the fenced writer lease, Litestream replication and restore;
• and every bug-hunt fix: #3564, #3565, #3586, #3587, #3617, #3690, #3691, #3693, #3696, #3736, #3737, and #3642–#3644 from swamp-uat#715.

The Datastore UAT is green on s3-v3 and gcs-v3.

This is the last Phase 4 item, I · Dogfood and benchmark: use solo v3 the way a team would, measure it against v2 and the kill criteria, and decide whether v3 on buckets can be announced. It changes no code in this issue. Problems it finds are filed as their own swamp-club issues.

Where it runs

• A new, throwaway swamp repo in a temp directory (mktemp -d), never a real project. One repo per datastore under test, plus a second checkout of each (a copy of the repo without .swamp/, as a teammate's clone).
• The released binary only, never a branch build. Use the newest release from gh release list --repo swamp-club/swamp. It must contain #3736 (97a6e9c7f) and #3737 (9bc473d7e). If no published release does yet, wait for one; don't dogfood an older one. Record the exact version.
• swamp init --tool none in the temp repo, so it never installs skills into ~/.claude. Remove the temp directories, buckets and containers when done.

What to run

1. The datastores, each set up with the released binary:
   • v3 filesystem: datastore setup filesystem --path <tmp>/ds --format 3;
   • v3 S3 and v3 GCS: datastore setup extension @swamp/s3-datastore|@swamp/gcs-datastore --config … --format 3. Use real test buckets for timings, and the ministack/fake-gcs emulators for the functional pass;
   • a v2 datastore of each kind, as the baseline, run with the same binary and the same steps.
2. Populate it with a realistic amount of data, through the CLI only:
   • a few hundred models (command/shell and a pulled extension type), definitions and workflows;
   • thousands of method runs and data versions, mixing inline records (< 8 KiB) and chunked content (KiBs to tens of MiB, including some > 16 MiB on GCS);
   • outputs made executable and 0600;
   • dot-named and non-ASCII data names;
   • multi-step workflows, including one killed mid-run (SIGKILL), then workflow recover;
   • deletes, renames, data gc, run gc and retention;
   • a namespaced repo beside a solo one;
   • managedConfig on and off;
   • swamp serve for a stretch, running workflows through it.
3. The second checkout:
   • a fresh checkout catches up;
   • both write in turn, with the lease handed over;
   • a checkout killed mid-write recovers;
   • a checkout holding v2 records is refused, and doctor datastores --repair works;
   • on buckets, a checkout whose .swamp/ is deleted restores from the replica.

What to measure

Against the v2 baseline on the same machine, and against the uat#496 benchmark lines. Report medians and p95, with an A/A control:

• Per-command latency:

  • a write (model method run);
  • a workflow step;
  • reads (data list, data get, data query);
  • first clone (catch-up) of the populated repo.

• The solo part of the kill criteria (design/enablers/datastore-commit-log.md, "Kill criteria", K2):

  • p95 flush (commit) latency, solo, at most 500 ms;
  • the commit adds at most 10% to median step wall time, compared with v2's pushChanged.

  Record whether either fires.

• Real-bucket timings (deferred from #3417), on real S3 and GCS:

  • cold and warm forced sync;
  • the idle sidecar's connection lifetime;
  • GCS uploads over 16 MB;
  • restore time and replica growth.

• Storage: bytes in the datastore for v3 vs v2 after the same work, and after data gc and compaction.

• macOS: the chunk barrier's cost after #3737, on an idle and a busy disk.

What to decide

1. Do the kill criteria fire? If one does, stop and report. Don't announce.

2. Confirm-before-output (9a): with --json, a v3 command can print success and then exit 75 when replication fails at close. Choose one, with the reason:

   • fix it before announcing (a follow-up issue);
   • or announce with it as a documented known limit.

   Recommend: announce with it documented, unless dogfooding shows it in practice.

3. Announce v3 on buckets: yes, or no with what blocks it.

Deliverables

• A results write-up (numbers, findings, decisions), linked from this issue and summarised in design/enablers/datastore-litestream.md (real-bucket timings) and the datastore rework plan.
• A swamp-club issue for every bug found, linked here. Don't fix them in this issue.
• Any new benchmark line worth keeping (for example a v3 bucket write) proposed for swamp-uat's benchmark suite as a filed swamp-uat issue, not added here.

Done when: every measurement above is recorded, the kill-criteria check and the 9a decision are written down, and the announce/no-announce call is made.

Blocked by: a published release that contains #3736 and #3737.