Claude Code gains plugin evals that score a skill against a no-plugin baseline
The new command runs realistic prompts with and without a plugin loaded, grades the output with six grader types, and can gate skills in CI.
Anthropic has published a plugin evaluation workflow for Claude Code, according to MarkTechPost. A new command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run in which the plugin is not loaded. The workflow is meant to answer three questions plugin developers previously could not measure: whether a skill triggers, whether it survives an edit or a new model, and whether it beats a bare model. It runs on Claude Code version 2.1.269 or later against any directory containing a plugin manifest or a skills-directory plugin. Every eval run and every judge grader is a real model call billed to the developer's plan or API account. An eval suite lives in an evals directory inside the plugin, with each case a subdirectory holding a prompt file and a graders folder. The prompt body is sent to Claude exactly as written, and path mentions are not expanded. Frontmatter on the prompt can set a turn limit, which defaults to 10, a timeout that defaults to 300 seconds, plus model, tags and allowed tools. Graders are markdown files whose frontmatter sets a type, an optional weight, and an optional arm. There are six grader types, four of which cost nothing to run. MarkTechPost says the workflow can serve as a continuous integration gate for skills.