Claude Code Plugin Eval Command: Full 2026 Guide

Wondering how the claude code plugin eval command works? The quick version is just below — and if you build plugins or skills for Claude Code, this is the feature you’ve been missing without realising it. Added in Claude Code v2.1.269 on 11 September 2026 (per the official changelog: “Added claude plugin eval: run a plugin’s eval suite against Claude Code and get scored, reproducible results (JSON + HTML report)”), it finally gives plugin authors what model builders have had for years: a proper eval harness. I’ve read the full official documentation, and here’s the complete picture.

What the Claude Code Plugin Eval Command Does

Per the official docs (fetched September 2026): “claude plugin eval runs your plugin against a suite of test cases and scores the results. Each case is a realistic prompt plus one or more graders. A grader is a pass/fail check on what Claude produced, such as a regex over the reply, whether a particular tool was called, or a rubric that a second model judges the reply against.”

For each run of a case, Claude Code starts a fresh, isolated, non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case’s turn or time limit. Each grader then checks the final reply, the transcript, or a file Claude created, and passes or fails.

Why bother? Three documented reasons: to measure how reliably your plugin steers Claude to the right outcome, to catch regressions when you change the plugin or a new model ships, and to see what the plugin contributes compared with no plugin at all. That last one is the killer feature, and it deserves its own section.

The Δ Column: Proving Your Plugin Actually Helps

A high score on its own doesn’t tell you the plugin helped, because Claude might do just as well without it. So by default, each case’s runs are repeated with no plugin loaded, and you get two scores — WITH and W/OUT — plus their difference, Δ, which is what the plugin contributed. The docs are blunt about it: “If a case scores 1.0 both with and without the plugin, the plugin isn’t what made it pass.”

Scoring works like this: one run of a non-deterministic agent tells you little, so each case runs three times by default. A run’s score is the fraction of its graders that passed (weighted if you set weights), a case’s score is the mean across runs, and a case passes when it meets --threshold — 1.0 by default. A typical summary line looks like:

CASE        WITH  W/OUT Δ      RUNS COST
first-case  1.00  0.33  +0.67  6    $0.41

The most common first finding, straight from the docs: a Δ near zero with the tool_used: Skill grader failing — meaning Claude isn’t choosing your skill on natural phrasing. The fix is usually rewriting the skill’s description, then re-running the suite to compare.

How to Use the Claude Code Plugin Eval Command Step by Step

You need Claude Code v2.1.269 or later (claude --version to check, claude update to upgrade), a plugin directory with a plugin.json or .claude-plugin/plugin.json manifest, and your normal Claude Code authentication — because eval runs and judge graders call the model with your credentials and count against your plan’s usage or API bill.

1. Generate the suite: from the plugin root, run claude plugin eval init. An interactive session opens in which Claude reads your plugin, asks what a good result looks like, proposes prompts that should and shouldn’t trigger it, designs graders, pilots them once, and writes one case directory per prompt under evals/. (Prefer to hand-write? claude plugin eval init --bare first-case gives you a blank template.)
2. Run it: claude plugin eval . runs every case — three times with your plugin, three times without, so one case is six runs, with a progress line per run.
3. Read the report: the summary table prints WITH, W/OUT, Δ, RUNS and COST (a list-price estimate), then writes a self-contained report.html under evals/results/<timestamp>/ showing every grader’s verdict and explanation — and for judge-scored graders, the judge’s votes and the excerpt it judged.
4. Iterate cheaply: claude plugin eval . --case <case-name> --runs 1 --ablation none runs a single arm once while you tune a case — then confirm any change at the default three runs before you trust it.

A case itself is just files: a prompt.md whose frontmatter sets limits like max_turns and allowed_tools and whose body is the user-style prompt, plus a graders/ directory where each Markdown file is one check — type: llm with a PASS/FAIL rubric, a regex, or a tool-use check.

Gating CI on Your Plugin’s Score

The documented CI pattern is tidy. Run with --json results.json for an archivable result document, pin both models so scores stay comparable, keep the report local and cap the spend:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

Then fail the build on the exit code: 0 means every case met the threshold, 1 means a case scored below it (or a config problem), 2 means a partial run — the --max-cost-usd ceiling was hit or a credential was rejected — with results.json still written and flagged partial: true, and 130 means interrupted. The JSON is a versioned document (schemaVersion: 1) with fields like aggregates.casesPassed and aggregates.meanDelta for your gating script.

Cost control matters here — the docs warn that every eval run and every judge grader is a real model call on your account. Between --max-cost-usd, --runs, and per-case turn limits, you can keep a suite affordable; the documented example run cost $0.41 for one case.

Where This Fits in the Claude Code Testing Story

Claude Code now has three distinct check layers for extension authors: claude plugin validate for syntax and schema errors in your plugin files, the skill-creator plugin’s own evals/evals.json flow for individual skills (the docs note the two case formats are separate), and now claude plugin eval for behavioural testing with a baseline. If you maintain skills, this pairs naturally with the Skill Doctor context audit — my Claude Code Skill Doctor guide covers that — and if your plugin ships MCP servers to a team, see my managed MCP servers setup guide. Model pinning for comparable scores is also easier now the default model is settled; my take on that is in the Claude Fable 5.1 review.

The Bottom Line on the Claude Code Plugin Eval Command

Shipped 11 September 2026 in v2.1.269, the claude code plugin eval command turns plugin quality from vibes into numbers: realistic prompts, pass/fail graders, three runs per case, a no-plugin baseline that exposes whether your plugin does anything at all, an HTML report you can share, and exit codes built for CI. If you ship plugins or skills to anyone but yourself, this should be in your release checklist from today. It’s the same shift software testing went through years ago — and the authors who adopt it first will ship noticeably more reliable plugins.

FAQ: claude code plugin eval command

What does the claude plugin eval command do?

It runs your Claude Code plugin against a suite of test cases — each a realistic prompt plus one or more pass/fail graders — in fresh, isolated, non-interactive sessions, scores the results, and produces a summary table plus JSON and HTML reports.

What version of Claude Code do I need for plugin evals?

Claude Code v2.1.269 or later, released 11 September 2026 per the official changelog. Run claude --version to check and claude update to upgrade.

How is a plugin eval scored?

Each case runs three times by default; a run’s score is the fraction of graders that passed, the case’s score is the mean across runs, and a case passes when its score meets --threshold (1.0 by default).

What is the no-plugin baseline in claude plugin eval?

Each case’s runs are repeated with no plugin loaded, giving WITH and W/OUT scores. Their difference, Δ, is what your plugin contributed — if a case scores 1.0 both ways, the plugin isn’t what made it pass.

Does claude plugin eval cost money?

Every eval run and every judge grader is a real model call on your account, counted against your plan’s usage or your API bill. The COST column is a list-price estimate, and --max-cost-usd caps a run’s spend.

How do I run claude plugin eval in CI?

Use --json for an archivable result, --trust-plugin to skip the trust prompt, pinned --model and --judge-model, --no-publish, a --threshold, and a --max-cost-usd ceiling — then fail the build on a non-zero exit code.

Related reading

• Claude Code Skill Doctor: Cut Wasted Context Fast
• Claude Code Managed MCP Servers: Full Setup Guide
• Claude Fable 5.1 Review: Worth the Upgrade?

Last updated September 2026. This is the living guide to the claude code plugin eval command — it gets updated as the tools change.