Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesskunk-ape

Relationships

#3667 Skill trigger evals run through claude -p so verify-skills works on a subscription login

Opened by skunk-ape · 10/20/2026· Shipped 10/20/2026

Problem

deno task eval-skill-triggers, the skill-trigger-eval step of the verify-skills workflow, calls the Anthropic Messages API directly through promptfoo, using ANTHROPIC_API_KEY from ~/.config/swamp/verify.env. The local key was revoked for excessive spend, so the step fails at preflight with an x-api-key request to api.anthropic.com/v1/messages. A Claude Code subscription token (claude setup-token) can't stand in for it: the API rejects that token as an x-api-key.

The verify-reviews workflow already runs claude -p, so it works on a claude.ai login.

There's also a second gap. Per scripts/setup_verify_env.ts, a missing ANTHROPIC_API_KEY makes the trigger evals skip with exit 0, so verification goes green having checked less.

Proposal

Run the eval's Anthropic models through the Claude Code CLI, so it uses the subscription login:

  • Keep promptfoo, the eval JSON files, the generated tests and the assertions.
  • Add a custom promptfoo provider that shells out to: claude -p "<query>" --model <id> --tools "" --strict-mcp-config --setting-sources "" --system-prompt "<router prompt + skill/guide descriptions>" --json-schema '<enum of tool names>' --output-format json It reads structured_output and returns it in the shape the existing assertions expect. Do not use --bare: it only accepts an API key.
  • Apply this to all three phases (trigger 84 cases, routing 52, sufficiency 51).
  • Replace the preflight x-api-key request with a claude -p ping.
  • When the CLI route is in use, a missing ANTHROPIC_API_KEY must no longer cause a silent skip. Update setup_verify_env.ts and its doc text to match.
  • Keep the API route for the GitHub multi-model-eval.yml (GPT and Gemini, plus Anthropic via the CI secret), for example with a flag or by choosing automatically.

Spike, 2026-10-20: one routing call through claude -p on Opus, with no API key set, returned the correct structured pick in about 3 s and about 1.2k input tokens.

Acceptance

  • verify-skills passes locally with no ANTHROPIC_API_KEY set, using only the claude.ai login.
  • Pass rates are re-baselined against the 0.9 threshold. Note any shift caused by moving from forced tool calls to a JSON-schema choice.
  • CI's multi-model eval is unchanged.
  • The config-integrity hashes and attestation (ci.yml) are updated for every changed script.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 10 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

10/20/2026, 4:53:50 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
skunk-ape assigned skunk-ape10/20/2026, 3:29:55 PM

Sign in to post a ripple.