#3724 Installers intermittently fail resolving stable release: add bounded retries and actionable diagnostics

Production Worker mitigation deployed — monitoring in progress

What was failing

The installer error was downstream of a real artifact-discovery failure. Cloudflare logs showed HEAD requests to stable archive URLs returning HTTP 403 while subsequent GETs succeeded. The latest pre-change 30-minute export contained 10 artifact HEAD 403s, 7 HEAD 499s, and one GET /install.sh 403. The 499s are separate observations, not established as the same cause.

For the Apple Silicon failure at 2026-10-21 17:00:56 UTC, trace 7d419c2ab473c692c47d0f1c384b9eb4 showed artifacts-edge-gcp fetching stable.json successfully, then receiving HTTP 403 from a HEAD to the version-pinned archive on [HOST-1] after 1.482 seconds. The selected release was 20261021.155838.0-sha.f979f7c1. The archive exists; current GCS checks found the Worker service account has objectViewer access and its HMAC key is active.

The shell installer ignores failed HEAD discovery, downloads the stable archive and checksum, and subsequently rejects the unresolved stable version with the misleading message: Could not determine the release version for checksum verification. This path was also reproduced deterministically offline.

Change deployed by the maintainer

At the end of signedGcsGet in the production artifacts-edge-gcp Worker, the upstream fetch now adds cache: no-store only for HEAD:

return await fetcher("https://" + host + canonicalUri, {
  method,
  headers,
  ...(method === "HEAD" ? { cache: "no-store" } : {}),
});

This bypasses the implicit Cloudflare fetch cache for signed HEAD requests to GCS. The existing explicit caches.default lookup, GET behavior, signing method, URL validation, and stable metadata remain unchanged. The maintainer deployed the change manually; it has not yet been committed to platform-swamp source.

Why this mitigation

GCS authentication signs the HTTP method. Cloudflare documents that cacheable HEAD requests can become GET requests on a cache miss. If that transformation occurs after HEAD signing, it would invalidate the signature. Disabling upstream fetch caching targets that possible interaction and fits the observed pattern of failed HEADs followed by successful GETs/cache hits.

This mechanism is still a hypothesis, not a proven root cause. The trace records the fetch as HEAD and does not contain the GCS rejection body or error code. Historical GCS Data Access logs were not enabled. A conditional upstream HEAD also succeeded before the change, so not every uncached HEAD was failing.

Validation so far

After deployment, conditional HEAD requests using the exact archive ETag bypassed the Worker local cache and returned 200 from the London edge:

• Pinned archive: 17:49:11 UTC, Ray a4e218f21a2ff37d-LHR.
• Stable archive: 17:49:18 UTC, Ray a4e2191c0fcccd11-LHR; required version-pinned redirect metadata present.

An initial diagnostic wildcard If-Match probe returned 412 at 17:48:59 UTC; this was our probe, not an installer failure.

The maintainer is monitoring Worker events containing 403 for ten minutes after deployment. Success means no new fetch 403 / HEAD 403 events while comparable HEAD traffic and successful upstream fetches continue; cache-hit successes alone are insufficient. The observation window is still pending, so this issue is not yet declared resolved.

Follow-up

Record the monitoring result and deployed Worker version. If effective, carry the exact mitigation into platform-swamp with focused HEAD/GET request-policy tests so redeployment preserves it. The installer improvements proposed in this issue—bounded retry/timeouts, early immutable release resolution, and accurate stage diagnostics—remain separate resilience work.

References: https://[HOST-2]/cache/concepts/cache-behavior/ and https://[HOST-2]/workers/runtime-apis/fetch/#parameters

Another occurrence: swamp-uat CI, Linux x86_64, 17:23:29 UTC

This was before the Worker mitigation deployed around 17:49. Same failure path as above:

• The stable archive's HEAD came back with no redirect metadata.

• The installer downloaded the stable alias URL anyway.

• The checksum request did follow the redirect (to 20261021.155838.0-sha.f979f7c1), and the checksum verified.

• release_version_from_url then rejected stable.

• Job: https://github.com/swamp-club/swamp-uat/actions/runs/39643330827 (attempt 1, "Remote Datastore Benchmarks (filesystem-v3)", step "Install swamp", on a GitHub-hosted ubuntu runner).

• Command: curl -fsSL https://artifacts.swamp-club.com/install.sh | sh -s -- -d "$HOME/.local/bin"

- Downloading https://artifacts.swamp-club.com/swamp/stable/binary/linux/x86_64/swamp-stable-binary-linux-x86_64.tar.gz to /tmp/tmp.W6Ti5JiBbv/swamp-stable-binary-linux-x86_64.tar.gz (curl)
- Downloading https://artifacts.swamp-club.com/swamp/stable/binary/linux/x86_64/swamp-stable-binary-linux-x86_64.tar.gz.sha256 to /tmp/tmp.W6Ti5JiBbv/swamp-stable-binary-linux-x86_64.tar.gz.sha256 (curl)
- Following S3 redirect to https://artifacts.swamp-club.com/swamp/20261021.155838.0-sha.f979f7c1/binary/linux/x86_64/swamp-20261021.155838.0-sha.f979f7c1-binary-linux-x86_64.tar.gz.sha256
- Verified SHA-256 checksum for 'swamp-stable-binary-linux-x86_64.tar.gz'

xxx Could not determine the release version for checksum verification

What this case adds:

• Mixed releases are possible. The archive came from the alias and the checksum from the pinned release. Here both were the same release, but if stable had moved between the two requests, the archive and checksum would belong to different releases. That is the "resolve once, pin both" item in the requested improvements.
• CI hits it too, not only interactive installs. The re-run of the same job passed.

Shipped: production Worker HEAD-cache mitigation

Marking shipped at the maintainer’s explicit request following their manual deployment to artifacts-edge-gcp.

The deployed change adds cache: "no-store" only to signed upstream HEAD requests in signedGcsGet. This bypasses Cloudflare fetch caching for those GCS requests while retaining the explicit Worker cache and existing GET behavior.

Confirmed evidence: production traces show GCS returning 403 to archive HEAD requests across macOS and Linux and multiple Cloudflare locations. The Linux request at 17:23:25.100 UTC (trace aa331baacf61fd789359ce09bbd62764, Microsoft ASN 8075, curl/8.5.0) is consistent with Seth’s CI report, though exact job attribution is not proven. All supplied failure traces ran the pre-mitigation Worker version. Post-deployment conditional HEAD checks returned 200 for both the pinned archive and stable alias at 17:49:11 and 17:49:18 UTC, with required stable metadata intact.

The hypothesized cause is cache-related HEAD-to-GET conversion invalidating method-bound GCS signatures. The underlying GCS rejection reason was not captured, so this remains unconfirmed. Shipping records the deployed mitigation and maintainer acceptance, not a claim of independently verified zero errors throughout the monitoring window.

Follow-up: reconcile the live patch into platform-swamp source with focused HEAD/GET tests. The original broader installer work (bounded retries/timeouts, resolve-once immutable pinning, and clearer failure diagnostics) was not implemented by this Worker patch and remains separate resilience work.