Relationships
#3667 Skill trigger evals run through claude -p so verify-skills works on a subscription login
Opened by skunk-ape · 10/20/2026· Shipped 10/20/2026
Problem
deno task eval-skill-triggers, the skill-trigger-eval step of the verify-skills workflow, calls the Anthropic Messages API directly through promptfoo, using ANTHROPIC_API_KEY from ~/.config/swamp/verify.env. The local key was revoked for excessive spend, so the step fails at preflight with an x-api-key request to api.anthropic.com/v1/messages. A Claude Code subscription token (claude setup-token) can't stand in for it: the API rejects that token as an x-api-key.
The verify-reviews workflow already runs claude -p, so it works on a claude.ai login.
There's also a second gap. Per scripts/setup_verify_env.ts, a missing ANTHROPIC_API_KEY makes the trigger evals skip with exit 0, so verification goes green having checked less.
Proposal
Run the eval's Anthropic models through the Claude Code CLI, so it uses the subscription login:
- Keep promptfoo, the eval JSON files, the generated tests and the assertions.
- Add a custom promptfoo provider that shells out to:
claude -p "<query>" --model <id> --tools "" --strict-mcp-config --setting-sources "" --system-prompt "<router prompt + skill/guide descriptions>" --json-schema '<enum of tool names>' --output-format jsonIt readsstructured_outputand returns it in the shape the existing assertions expect. Do not use--bare: it only accepts an API key. - Apply this to all three phases (trigger 84 cases, routing 52, sufficiency 51).
- Replace the preflight
x-api-keyrequest with aclaude -pping. - When the CLI route is in use, a missing
ANTHROPIC_API_KEYmust no longer cause a silent skip. Updatesetup_verify_env.tsand its doc text to match. - Keep the API route for the GitHub
multi-model-eval.yml(GPT and Gemini, plus Anthropic via the CI secret), for example with a flag or by choosing automatically.
Spike, 2026-10-20: one routing call through claude -p on Opus, with no API key set, returned the correct structured pick in about 3 s and about 1.2k input tokens.
Acceptance
- verify-skills passes locally with no
ANTHROPIC_API_KEYset, using only the claude.ai login. - Pass rates are re-baselined against the 0.9 threshold. Note any shift caused by moving from forced tool calls to a JSON-schema choice.
- CI's multi-model eval is unchanged.
- The config-integrity hashes and attestation (
ci.yml) are updated for every changed script.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.