Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#2211 validate-attestation hashes the PR merge commit, so a moving base invalidates valid attestations

Opened by skunk-ape · 9/16/2026· Shipped 9/17/2026

The validate-attestation job compares configIntegrity checksums recorded at the PR head commit against files checked out from the PR merge commit. Whenever the base branch moves in a way that touches one of the hashed files between attesting and CI running, a perfectly valid attestation fails.

Observed

PR #2475 (swamp-club#2199), a two-file test-only change:

  • Attestation built at head 93d7d825, whose base was 245276a1. AGENTS.md there hashes to [REDACTED-SECRET-1].
  • PR #2474 (b07863c9) merged to main at 20:28 UTC and modified AGENTS.md.
  • CI ran at 21:15 UTC and hashed AGENTS.md as [REDACTED-SECRET-2].
  • Mismatch → errors=1 → job exits 1.

Every other check passed: the attestation was fetched successfully, was valid JSON in the expected shape, subject.commit matched the PR head, the gate was allPassed: true with 19/22 steps, and freshness was about four minutes.

Cause

.github/workflows/ci.yml:483:

- name: Checkout code
  uses: actions/checkout@v6

With no ref:, actions/checkout on a pull_request event checks out refs/pull/N/merge — the head merged into the current base. The check_hash calls further down then sha256sum those merged working-tree files, while the attestation recorded hashes of the files as they existed at the head commit.

The two only agree when the base has not touched any hashed path since the attestation was produced. The hashed set is broad and frequently edited: CLAUDE.md, AGENTS.md, verification/review-prompts/*.md, verification/workflow-verify-*.yaml, scripts/review_skills.ts, evals/promptfoo/package.json.

Impact

Any PR can fail this gate through no fault of its own, and the failure is non-deterministic — it depends entirely on what merged to main in the minutes between running local verification and CI picking up the PR. The remedy today is to rebase, re-run all three verification workflows, and post a fresh attestation, which is itself subject to the same race.

This also weakens the guarantee the check is meant to provide. The intent is "the config used during verification matches the config at the verified commit". Hashing the merge commit answers a different question — "does the verified config match the base branch right now" — which the attestation was never claiming.

Suggested fix

Pin the comparison to the attested commit. Either:

  1. Check out the head explicitly in this job:

    - uses: actions/checkout@v6
  with:
    ref: ${{ github.event.pull_request.head.sha }}

  2. Or leave the checkout alone and hash the blobs at the attested commit instead of the working tree, e.g. git show "${att_commit}:${file}" | sha256sum, which additionally verifies that the attestation's own commit is the one being hashed.

Option 2 is the stronger of the two, since it ties the hash comparison to subject.commit rather than to whatever the runner happens to have checked out.

Notes

Found while shipping swamp-club#2199. Filed alongside a companion issue about this failure producing no log output, which made it considerably harder to diagnose than it should have been.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPEDTRIAGE+ 5 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/17/2026, 12:21:28 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/16/2026, 10:47:49 PM
Editable. Press Enter to edit.

stack72 commented 9/17/2026, 12:22:06 AM

Thanks @skunk-ape for reporting this! We shipped: Fix validate-attestation CI job by pinning its checkout to the PR head commit instead of the default merge ref. Add ref set to github.event.pull_request.head.sha to the actions/checkout step. This makes the working tree match the commit the attestation was built at, eliminating the hash mismatch when the base branch moves. . The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.