#3656 Registry client: retry transient upstream failures with bounded backoff and visible attempt progress

Request

Make swamp recover gracefully from transient upstream registry failures instead of failing extension resolution/setup after one timeout. This is core CLI HTTP behavior, not a gcs-datastore extension defect.

Show useful progress, for example: Registry request timed out; retry 1/5 in 1s. Define the count clearly: an initial attempt plus at most five retries. Suggested waits: 1s, 2s, 4s, 8s, 16s, with jitter and a bounded total time budget. Preserve prompt cancellation and existing Retry-After handling. Progress belongs on stderr/structured logs so JSON stdout stays valid.

Reproduction and evidence

On ops.swamp-club.com, run isolated CLI-6 and datastore-gcs-2 shards against GCP-built swamp 20261020.101849.0-sha.26dc044f (source [REDACTED-SECRET-1]), UAT commit [REDACTED-SECRET-2]. Five sequential runs on fresh GCP VMs: CLI-6 passed 5/5; GCS-2 passed 4/5.

Failure run: 2f021d9c-dfa0-4abd-b20c-7599a42c93a2. Test: swamp workflow approve leaves nothing unsynced and reaches a second and a fresh clone [managed-config] [gcs]. A fresh clone runs swamp datastore setup extension @swamp/gcs-datastore --config .... Storage uses a local GCS emulator, but extension auto-resolution contacts the real https://swamp-club.com registry. Setup fails before verification of the fresh clone with:

Error: Datastore type "@swamp/gcs-datastore" is not registered.
auto-resolution failed for type @swamp/gcs-datastore: Request to https://swamp-club.com timed out. Is the server running?

The same registry timeout affected three GCS tests in the preceding full 36-shard release run d7b81ad0-10a9-4c26-8206-00565d9c2bad. Equivalent GitHub datastore tests passed.

At this exact source revision, src/infrastructure/http/extension_api_client.ts uses AbortSignal.timeout(15_000). src/infrastructure/http/rate_limit.ts retries eligible HTTP 429 responses, but a thrown timeout/connection error escapes without retry. The current timeout diagnostic omits the request path.

Axiom production evidence for 2026-10-20 12:20–12:25 UTC: recorded gcs-datastore requests had 335 HTTP 200 and 84 HTTP 302 responses; HTTP-200 p95 handler duration 45ms, maximum 1183ms. No recorded API 429/5xx or handler duration >=5s in that window. This does NOT establish where the failed request stalled: DNS/TLS/network/edge/client scheduling remain unconfirmed, and requests not reaching the application are absent from these logs.

Acceptance criteria

• Retry safe idempotent registry reads on transient connection resets/timeouts and selected transient HTTP failures (e.g. 502/503/504), with bounded attempts, exponential backoff/jitter, and per-attempt plus total deadlines.
• Preserve existing 429 Retry-After semantics under a single bounded retry policy; avoid multiplying nested retry budgets.
• Do not automatically replay writes/uploads/publishes without an explicit idempotency guarantee. Do not retry authentication/permission/not-found/validation failures or explicit cancellation.
• Show retry N/5, reason and wait; final error includes sanitized method/path, elapsed time, attempts and underlying cause. Never expose credentials or signed query parameters.
• Preserve atomic installation and checksum verification; clean incomplete downloads before retrying.
• Deterministic injected-fetch/clock tests cover timeout then success, transient HTTP failure then success, exhausted retries, abort during request/backoff, and nonretryable errors. Include an auto-resolution setup regression test.

This request improves resilience; it does not claim to fix the still-unproven underlying network/client scheduling cause. Simply raising test timeouts or retrying entire workflows would hide the dependency-fetch failure rather than handling it at the appropriate boundary.