#3191 Datastore rework Phase 3: measure and bound the format-guard marker read's cost

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865; design: design/enablers/datastore-commit-log.md). Phase 3 adds a v3 datastore format that a user can opt in to for a new, empty datastore. Before any v3 code ships, released binaries must refuse a datastore marked with a newer format. That format guard shipped in swamp-club#3189 (main at a7ab3774).

How the guard works today:

• The marker. It is <path>/datastore-format.json on filesystem datastores and the control-plane record _control/datastore-format on S3 and GCS, holding {format, minReaderFormat, writtenBy}. No marker means format 2.
• Where it runs. It runs at resolveDatastoreForRepo, in acquireModelLocks, in datastore setup and for serve audit stores. It always runs before any lock, pull or push.
• The read. readDatastoreFormatMarker in src/infrastructure/persistence/datastore_format_marker_reader.ts:
  1. resolves the provider;
  2. calls provider.createSyncService(...) twice. The first instance is used for the read. The second only checks that the provider doesn't share one instance;
  3. makes one controlPlaneStore().get("datastore-format"), raced against DATASTORE_FORMAT_READ_TIMEOUT_MS = 5_000.
• Failed reads. A read that fails or times out proceeds, as today, with a debug log. Errors are datastore_format_unsupported and datastore_format_marker_invalid.
• Why a fresh service. The S3 and GCS extensions bind their namespace lazily, on the first pull or push (bindNamespace in s3_cache_sync.ts and gcs_cache_sync.ts in swamp-extensions). A fresh, never-pulled service therefore reads the datastore-wide record _control/datastore-format even when the repo has a namespace configured. The design doc records this. It also records that v3 must read the marker at a datastore-scoped key instead.

The gap. Item 7 of #3189's design asked for the added latency to be measured and reported. It was never measured. Every S3 or GCS command now pays for:

• resolving the provider;
• building two extra sync services, which means client construction and credential resolution;
• one round trip to the bucket.

A slow or unreachable bucket can add up to about 5 s before the command proceeds. Extension retries inside get happen inside that window. We don't know the healthy cost.

Goal

Know what the guard costs, and bound it. The v3 opt-in will make the marker read heavier, so this should be settled before then.

Design

1. Measure first, before changing any code. Run against ministack (S3) and fake-gcs (GCS):

   • Commands: data list, model method run and datastore sync.
   • Conditions: a healthy emulator, then a paused one (docker pause).
   • Comparison: the guard against DATASTORE_FORMAT_READ_TIMEOUT_MS patched to skip the read. Run at least 20 times each.
   • Report: median, p95 and the extra request count.
   • Where: use the swamp-uat performance baseline (swamp-uat#496) if it exists, otherwise a local run with the script committed under scripts/ or documented in the PR.

2. Datastore-scoped marker read. Add an explicit way to read the datastore-wide marker that doesn't depend on lazy namespace binding. For example:

   • an optional datastoreControlPlaneStore(), or a scope: "datastore" option, on the sync service or provider;
   • or a documented key both sides use.

   Keep the fresh-service path as the fallback for extension versions that don't offer it. Coordinate the extension side in swamp-extensions with an issue or PR linked from this one. v3 needs this read anyway.

3. Stop building services the read doesn't need. Drop the second createSyncService call used only for the identity check, or replace it with a cheaper check. Where the command will build its own sync service anyway, reuse that client for the marker read, as long as the read still happens before any lock, pull or push.

4. Bound the degraded path:

   • shorten the marker-read cap well below 5 s (pick it from the measurements; about 1–2 s is the expected range);
   • ask the extension not to retry this read, for example with an AbortSignal passed to get if the control-plane API takes one, or by adding one;
   • the read still proceeds on failure, unchanged.

5. Behaviour. Nothing changes apart from timing and request counts. The same markers refuse, the same error codes and messages appear, and every entry point is still guarded before anything writes.

Tests

• Existing tests unchanged. The existing format-guard unit and integration tests pass with no expectation changes, in particular the "nothing written" fingerprint tests.
• Request count. On the in-memory remote with a control plane, assert how many sync services and control-plane gets one guarded open makes. Record the count on the current code first, then show it drops.
• Bounded wait. A control-plane get that never resolves is abandoned at the new cap and the command proceeds. Use a fake clock or an injected timeout, with no real sleeps (see AGENTS.md timing rules).
• Namespaced read. A namespaced client finds the datastore-wide marker through the new datastore-scoped read, and through the fallback path. Check against ministack and fake-gcs, not by reasoning.
• Benchmark. Add the marker read to the swamp-uat#496 baseline, or file a swamp-uat issue for it and link it.

Out of scope

• Writing the marker. v3 writes it (Phase 3 opt-in) and the Phase 6 migration writes it.
• Caching the result across processes. A datastore can be migrated between runs.
• The lockfile publish work (Phase 3, "Lockfile publish as a unit commit"). It runs in parallel and touches different files.

Done when

• The PR description has the before and after numbers (median, p95, request count, degraded-path wait).
• The degraded path is bounded well under 5 s.
• The datastore-scoped read exists in core, with the extension side filed or merged.
• No behaviour test changed.

Blocked by: nothing (#3189 has merged).