#3803 v3 first clone and restore of a bucket datastore take 2.6-3x v2, missing #3418's 1.5x target (found by #3756)

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished solo mode: v3 datastores on filesystem, S3 and GCS, with one writer at a time. A new checkout of a v3 bucket datastore (a first clone), or one whose .swamp/ was deleted (a restore), restores the Litestream replica of the commit log, catches its read model up, and fetches content chunks from _blobs/. swamp-club#3418 set the target for this: within 1.5x of v2 on the same bucket kind.

The Phase 4 dogfood (swamp-club#3756) measured it on main at bc388c720 against ministack and fake-gcs. It misses the target by a wide margin.

Measurements

~/dogfood-3756/clone_restore.sh, 5 rounds per cell, interleaved, with A/A twins agreeing within 0.97–1.01. Item counts matched the source repo in all 60 samples.

• Clone: a new checkout (the repo files without .swamp/, a fresh SWAMP_HOME), timed until its data is local.
• Restore: the same checkout with .swamp/ and the swamp home's repos/ cache deleted.
• First command: on v3, data list blob5, which restores and catches up. On v2, datastore sync --pull then data list blob5, since v2 reads never pull.

The repos (s32/s33 on S3, gcs2/gcs3 on GCS, v2/v3) were populated by the same script: 200 command/shell models, 40 blob models, 10 @swamp/git models, 30 workflows, about 800 method runs and 150 workflow runs, blob sizes from under 8 KiB to over 17 MiB, managedConfig on. Local state is about 560–860 MB per checkout.

On the other side of the trade, every workflow step afterwards is far faster on v3 (about 60 ms against 1.1–1.3 s on v2), and writes and reads are about the same.

What is not known

The extra ~25 s has not been broken down. Large parts of the v3 restore path have no spans: in traces, about 22 s sat inside swamp.cli with no named child span. Candidates, none measured:

1. Chunk fetches to materialise every record locally: serial versus parallel, and per-chunk overhead. v2 pulls a similar byte volume into its cache.
2. The Litestream replica restore. On GCS the replica holds 1,438 flat L0 ltx objects, while S3's is compacted into levels (47 objects). Is compaction not running on GCS, or not running because sidecars are short-lived?
3. Listing _control/lineage/, which gains one head object per writing command (1,236 objects after about 1,250 commands). Is anything pruning old lineages, and does a clone list them all?
4. Replaying the commit log into the read model and the catalog.

Auto-installing the datastore extension is paid by both formats, so it can't explain the gap.

Work

1. Spans first. Add spans around the replica restore, the lineage listing, read-model catch-up and the chunk fetches (with counts and bytes as attributes), then break down a clone and a restore of the populated repos on both bucket kinds. Post the breakdown on this issue before changing anything else.
2. Fix what the breakdown shows, to reach the 1.5x target. If it shows the target can't be reached without a design change (for example, fetching content lazily instead of materialising everything at clone), stop and bring the options back with numbers rather than building one.
3. Answer the two storage questions (GCS L0 compaction, lineage pruning) in the PR, whether or not they turn out to cost clone time. If lineage objects are never pruned, prune them as part of this work.

Done when

• The spans exist and stay: a traced clone or restore names where its time goes.
• On the dogfood's repos, clone and restore are within 1.5x of v2 on S3 and on GCS (median of 5 interleaved rounds, with an A/A twin), or the issue holds a measured case for a design change and the user has decided on it.
• No regression in the quiet benchmark (~/dogfood-3756/bench.sh) for writes, reads and workflow steps.
• Tests for any behaviour change (parallel fetch limits, pruning) red on main first.

Severity: high for the announcement. A team's first checkout on v3 takes 2.6–3x as long as on v2. Blocks announcing v3 on buckets (swamp-club#3756).

Blocked by: nothing. Runs in parallel with the identity and outage-errors issues filed alongside it.