evaluate-plugin-batch
/evaluate-plugin-batch
Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.
- Author
- laurigates ↗
- Repository
- claude-plugins ↗
- Licence
- MIT
- Last repository change
- 24.10.2026
Assessment and security
- Files in this package
- 2
- Scripts
- 1
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 24.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 1 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: workflows/evaluate-plugin-batch.workflow.js
What it does and when to use it
Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.
Arguments
/evaluate-plugin-batch git-plugin [--create-missing-evals]
Original author description
Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.
How it works
- When to Use This Skill
- Context
- Parameters
- Workflow harness (template)
- Execution
- Agentic Optimizations
- Quick Reference
SKILL.md preview
---
name: evaluate-plugin-batch
description: Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.
args: <plugin-name> [--create-missing-evals] [--parallel N] [--harness subagent|headless]
allowed-tools: Task, Read, Write, Glob, Grep, Bash(bash *), SlashCommand
argument-hint: "git-plugin [--create-missing-evals]"
agent: general-purpose
created: 2026-03-04
modified: 2026-10-05
compatibility: claude-code
reviewed: 2026-09-02
---
# /evaluate:plugin-batch
Batch evaluate all skills in a plugin. Runs `/evaluate:skill` for each skill, then produces a plugin-level quality report.
## When to Use This Skill
| Use this skill when... | Use alternative when... |
|------------------------|------------------------|
| Auditing all skills in a plugin before release | Evaluating a single skill -> `/evaluate:skill` |
| Establishing quality baselines across a plugin | Viewing past results -> `/evaluate:report` |
| Checking overall plugin quality after refactoring | Need structural compliance -> `plugin-compliance-check.sh` |
## Context
- Available plugins: !`find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'`
## Parameters
Parse these from `$ARGUMENTS`:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `<plugin-name>` | required | Name of the plugin to evaluate |
| `--create-missing-evals` | false | Generate evals for skills that lack them |
| `--parallel N` | 1 | Max concurrent skill evaluations |
| `--harness subagent\|headless` | `subagent` | Rollout harness forwarded to every `/evaluate:skill` run (see that skill). One harness per batch, so the per-skill benchmarks stay comparable |
## Workflow harness (template)
`workflows/evaluate-plugin-batch.workflow.js` ships beside this skill. **It is a
TEMPLATE to adapt, not a script to run verbatim.** Read it, then rewrite it for
the work in front of you.
**Adapt freely:** the inventory and aggregation agent prompts, the effort tiers,
the `DEFAULT_WAVE` starting width, the `CELL_CAP` handed to each child, the
`skipped[]` reason strings, and the shape of the `rows` your project's report
actually needs.
**Preserve across any adaptation:** (a) the loop bound comes from
`inspect_eval.sh --plugin-dir <plugin>`'s `=== SKILLS ===` / `=== EVALS ===`
listings, never from a prose "for each skill" — and eval-readiness stays the
pure `s.hasEvals || createMissingEvals` filter over a file that exists or does
not, so no agent classifies it; (b) `REPORT_SCHEMA`'s closed
`complete | partial-sweep | aggregate-failed` status enum and its one-value
`denominatorSource` enum, which together make it structurally impossible to
report a plugin pass rate without stating that the denominator came from the
script; (c) the Aggregate stage is a barrier — `aggregate_benchmark.sh` walks
the filesystem for every skill's `eval-results/benchmark.json`, so its
denominators are only correct once the last cell has finished writing, and no
single cell can see the plugin-level numbers. The `harness` arg is forwarded
verbatim to every `workflow('evaluate-skill', …)` child — never mixed within one
batch, because subagent and headless pass rates are not comparable.
Three consequences of (a)–(c) that are also non-negotiable:
- **`CAP` (default 25) ABORTS; it never truncates.** A silently-shortened sweep
publishes a pass rate over a denominator nobody chose — which is precisely the
number this skill exists to produce correctly. The abort states the count and
the cap.
- **`--parallel N` is the caller's and is never superseded.** The wave width is
derived from `args.parallel`; `DEFAULT_WAVE` applies only when the caller
supplied nothing, and a non-integer value falls back loudly via `log()`. The
platform separately caps concurrency at `min(16, CPUs-2)`, and a nested child
shares this run's cap and agent counter.
- **The workflow's own count is only a cross-check.** `crossCheck.agrees`
compares the cells this run dispatched against the benchmarks the script found
on disk. A disagreement emits `partial-sweep` — that disagreement is the
anti-laziness signal, not a rounding error, and must never be smoothed into
`complete`.
**This harness runs near-empty until a golden set of `evals.json` files exists.**
Per [`.claude/rules/skill-evaluation.md`](../../../.claude/rules/skill-evaluation.md)
eval coverage is deliberately scoped to ~15–25 canary skills, so most plugins
today have zero or one eval-ready skill and the run aborts with
`reason: 'below-floor'`. That is the designed outcome, not a bug — an empty
sweep means the evals have not been written yet, and the fix is to author them
(or pass `--create-missing-evals`), never to widen the harness.
**Agent budget:** 2 + skills x evaluate-skill budget — the inventory and aggregate
agents, plus one `evaluate-skill` child per included skill (at most 25), each
spending that skill's own budget. The scale guard asks before every run: a
`workflow()` child's agents are never counted.
**Skip the harness when:** the plugin has fewer than 2 eval-ready skills — the
modal case today, and `FLOOR = 2` is a hard bound that aborts there rather than
a tunable knob, because a single skill is exactly what `/evaluate:skill` already
does without an inventory agent, a nested workflow and an aggregate agent on
top. The steps below remain the authoritative description of *what* each stage
must produce; the harness only fixes *how* the work is split.
`context: fork` stays **off** for this skill. The fan-out here is a caller-chosen
`--parallel N` width, so the `[1m]` concurrent-subagent cascade hazard in
[`.claude/rules/skill-fork-context.md`](../../../.claude/rules/skill-fork-context.md)
applies; `scripts/plugin-compliance-check.sh` keeps this skill out of its
`context: fork` pin list for that reason, and the harness does not change it.
## Execution
### Step 1: Discover skills
Find all skills in the plugin:
```
<plugin-name>/skills/*/SKILL.md
```
List them and count the total.
### Step 2: Filter and prepare
For each skill, check if `evals.json` exists:
- **Has evals**: include in evaluation
- **No evals + `--create-missing-evals`**: include, will create evals during evaluation
- **No evals, no flag**: skip with a note
Report the breakdown:
```
Found N skills in <plugin-name>:
- M with eval cases
- K without eval cases (skipped | will create)
```
### Step 3: Run evaluations
For each included skill, invoke `/evaluate:skill` via the SlashCommand tool:
```
SlashCommand: /evaluate:skill <plugin-name>/<skill-name> [--create-evals] [--harness <h>]
```
Forward `--harness` unchanged to every skill when it was given; omit it otherwise
(the default is `subagent`).
If `--parallel N` is set and N > 1, batch evaluations into groups of N. Otherwise, run sequentially.
Report progress from the results themselves: after each `/evaluate:skill`
returns, state which skill finished and its pass rate from `benchmark.json`.
(Todo/task tools are not available on Opus 4.8+/Sonnet 5/Fable-generation
models unless `CLAUDE_CODE_ENABLE_TODO_TOOLS=1` is set; do not depend on them
— see `.claude/rules/agentic-permissions.md` § Task-tool availability.)
### Step 4: Aggregate plugin report
After all skill evaluations complete, read each skill's `benchmark.json` and aggregate:
```
bash evaluate-plugin/scripts/aggregate_benchmark.sh <plugin-name>
```
Write aggregated results to `<plugin-name>/eval-results/plugin-benchmark.json`.
### Step 5: Report
Print a plugin-level summary table:
```
## Plugin Evaluation: <plugin-name>
| Skill | Evals | Pass Rate | Status |
|-------|-------|-----------|--------|
| skill-a | 4 | 100% | PASS |
| skill-b | 3 | 67% | PARTIAL |
| skill-c | 5 | 80% | PASS |
**Overall**: 82% pass rate across N eval cases
```
Rank skills by pass rate. Flag any below 50% as needing attention.
## Agentic Optimizations
| Context | Command |
|---------|---------|
| Inventory plugin skills + evals | `bash evaluate-plugin/scripts/inspect_eval.sh --plugin-dir <plugin>` |
| Inspect a single skill's evals | `bash evaluate-plugin/scripts/inspect_eval.sh --plugin <plugin> --skill <skill>` |
| Aggregate results | `bash evaluate-plugin/scripts/aggregate_benchmark.sh <plugin>` |
## Quick Reference
| Flag | Description |
|------|-------------|
| `--create-missing-evals` | Generate eval cases for skills without them |
| `--parallel N` | Max concurrent evaluations (default: 1) |
| `--harness headless` | Run every skill's rollouts through real `claude -p` children (default: `subagent`) |
Files in this package (2)
- SKILL.md 8 KB
- workflows/evaluate-plugin-batch.workflow.js 16 KB
Versions
The database has no other skills named evaluate-plugin-batch.
Database snapshot: 13.11.2026 · package version deef6b4d00c5 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100evaluate-plugin
Evaluate plugin quality. Use when user says \"evaluate plugin\", \"review plugin quality\", \"score my plugin\", \"check plugin\", \"rate plugin\". - 100check
Run every available check against a Claude Code plugin — the first-party validator, plugin-dev's linters, and three checks nothing else performs. Use when asked to validate, check, audit, or review a plugin, or before publishing one. - 100local-evaluate-plugin-cf170bc7
Evaluate a local Codex plugin in engineer-friendly language. Use when the user says "evaluate this plugin", "audit this plugin", "why did this score that way", "what should I fix first", "help me benchmark this plugin", or asks for a plugin-wide report before comparing versions. - 100plugin-review
Review plugin quality with tiered checks and dependency scoping. Use for PR and pre-release audits. - 100plugin-builder
Use when hardening, converting, auditing, or pre-release checking a Codex plugin package by verifying manifest paths, bundled skills, hooks, MCP/app config, validation gates, and release blockers. - 95evaluate-report
View evaluation results and benchmark reports for a skill or plugin. Use when reviewing past eval results, comparing benchmark runs, or tracking quality trends.