Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#1870 Move skill review and trigger eval into the local verification loop

Opened by stack72 · 8/27/2026

Problem

The skill review (deno run review-skills) and skill trigger eval (deno run eval-skill-triggers) currently run only in CI as separate jobs, gated by the skills path filter. They are not part of the local verification loop, which means:

  1. Skill review failures are discovered in CI, not before the PR opens
  2. The attestation doesn't include skill review results
  3. Config integrity checksums don't cover skill files beyond CLAUDE.md

Proposed Solution

Add skill checks to the local verification workflows:

Build workflow (verify-build)

Add two steps to the build workflow, gated by a file-change guard on skill paths (.claude/skills/**, CLAUDE.md, scripts/review_skills.ts, evals/promptfoo/**):

  1. skill-review — runs deno run review-skills (requires TESSL_TOKEN)
  2. skill-trigger-eval — runs deno run eval-skill-triggers (requires ANTHROPIC_API_KEY)

Both use command/shell steps with workingDir pointing at the verification worktree.

Attestation schema

Add skill check results to the attestation:

  • New steps in the steps array: {"job": "skills", "step": "skill-review", ...} and {"job": "skills", "step": "skill-trigger-eval", ...}
  • Guard status: skipped when no skill files changed (same as review guards)

CI attestation validator

Update validate-attestation to expect skill steps in the attestation when the PR touches skill paths.

Config integrity

Add checksums for:

  • scripts/review_skills.ts
  • evals/promptfoo/package.json (pins the promptfoo version used for trigger evals)

Authentication

The skill checks need API keys:

  • TESSL_TOKEN for skill review — source from ~/.config/swamp/verify.env (same file that provides ANTHROPIC_API_KEY)
  • ANTHROPIC_API_KEY for trigger evals — already available from verify.env or claude.ai login

Blocked by

The @swamp/git extension's files array is only populated when nameOnly: true is set. The guard expression for skill changes needs the file list from the changed-files step in the reviews workflow. This is the same limitation that affects other guard expressions — tracked separately.

Alternatives

  • Keep skill checks in CI only — but this breaks the attestation model where all checks should run locally and CI validates
  • Run skill checks unconditionally (no guard) — wasteful, the trigger eval calls the Anthropic API and costs money on every run even when no skills changed
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/27/2026, 5:02:04 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.