Skip to content
Skills Library
AI & agents · Review & audit

evaluate-plugin-batch

/evaluate-plugin-batch

Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.

Licence
MIT
Last repository change
24.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 24.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: workflows/evaluate-plugin-batch.workflow.js

What it does and when to use it

Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.

Arguments

/evaluate-plugin-batch git-plugin [--create-missing-evals]
Original author description

Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.

How it works

  1. When to Use This Skill
  2. Context
  3. Parameters
  4. Workflow harness (template)
  5. Execution
  6. Agentic Optimizations
  7. Quick Reference
Original SKILL.md headings · 185 lines, 1,162 words
SKILL.md preview
---
name: evaluate-plugin-batch
description: Batch evaluate every skill in a plugin and produce a plugin-level report. Use when auditing an entire plugin's quality or validating before a release.
args: <plugin-name> [--create-missing-evals] [--parallel N] [--harness subagent|headless]
allowed-tools: Task, Read, Write, Glob, Grep, Bash(bash *), SlashCommand
argument-hint: "git-plugin [--create-missing-evals]"
agent: general-purpose
created: 2026-03-04
modified: 2026-10-05
compatibility: claude-code
reviewed: 2026-09-02
---

# /evaluate:plugin-batch

Batch evaluate all skills in a plugin. Runs `/evaluate:skill` for each skill, then produces a plugin-level quality report.

## When to Use This Skill

| Use this skill when... | Use alternative when... |
|------------------------|------------------------|
| Auditing all skills in a plugin before release | Evaluating a single skill -> `/evaluate:skill` |
| Establishing quality baselines across a plugin | Viewing past results -> `/evaluate:report` |
| Checking overall plugin quality after refactoring | Need structural compliance -> `plugin-compliance-check.sh` |

## Context

- Available plugins: !`find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'`

## Parameters

Parse these from `$ARGUMENTS`:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `<plugin-name>` | required | Name of the plugin to evaluate |
| `--create-missing-evals` | false | Generate evals for skills that lack them |
| `--parallel N` | 1 | Max concurrent skill evaluations |
| `--harness subagent\|headless` | `subagent` | Rollout harness forwarded to every `/evaluate:skill` run (see that skill). One harness per batch, so the per-skill benchmarks stay comparable |

## Workflow harness (template)

`workflows/evaluate-plugin-batch.workflow.js` ships beside this skill. **It is a
TEMPLATE to adapt, not a script to run verbatim.** Read it, then rewrite it for
the work in front of you.

**Adapt freely:** the inventory and aggregation agent prompts, the effort tiers,
the `DEFAULT_WAVE` starting width, the `CELL_CAP` handed to each child, the
`skipped[]` reason strings, and the shape of the `rows` your project's report
actually needs.

**Preserve across any adaptation:** (a) the loop bound comes from
`inspect_eval.sh --plugin-dir <plugin>`'s `=== SKILLS ===` / `=== EVALS ===`
listings, never from a prose "for each skill" — and eval-readiness stays the
pure `s.hasEvals || createMissingEvals` filter over a file that exists or does
not, so no agent classifies it; (b) `REPORT_SCHEMA`'s closed
`complete | partial-sweep | aggregate-failed` status enum and its one-value
`denominatorSource` enum, which together make it structurally impossible to
report a plugin pass rate without stating that the denominator came from the
script; (c) the Aggregate stage is a barrier — `aggregate_benchmark.sh` walks
the filesystem for every skill's `eval-results/benchmark.json`, so its
denominators are only correct once the last cell has finished writing, and no
single cell can see the plugin-level numbers. The `harness` arg is forwarded
verbatim to every `workflow('evaluate-skill', …)` child — never mixed within one
batch, because subagent and headless pass rates are not comparable.

Three consequences of (a)–(c) that are also non-negotiable:

- **`CAP` (default 25) ABORTS; it never truncates.** A silently-shortened sweep
  publishes a pass rate over a denominator nobody chose — which is precisely the
  number this skill exists to produce correctly. The abort states the count and
  the cap.
- **`--parallel N` is the caller's and is never superseded.** The wave width is
  derived from `args.parallel`; `DEFAULT_WAVE` applies only when the caller
  supplied nothing, and a non-integer value falls back loudly via `log()`. The
  platform separately caps concurrency at `min(16, CPUs-2)`, and a nested child
  shares this run's cap and agent counter.
- **The workflow's own count is only a cross-check.** `crossCheck.agrees`
  compares the cells this run dispatched against the benchmarks the script found
  on disk. A disagreement emits `partial-sweep` — that disagreement is the
  anti-laziness signal, not a rounding error, and must never be smoothed into
  `complete`.

**This harness runs near-empty until a golden set of `evals.json` files exists.**
Per [`.claude/rules/skill-evaluation.md`](../../../.claude/rules/skill-evaluation.md)
eval coverage is deliberately scoped to ~15–25 canary skills, so most plugins
today have zero or one eval-ready skill and the run aborts with
`reason: 'below-floor'`. That is the designed outcome, not a bug — an empty
sweep means the evals have not been written yet, and the fix is to author them
(or pass `--create-missing-evals`), never to widen the harness.

**Agent budget:** 2 + skills x evaluate-skill budget — the inventory and aggregate
agents, plus one `evaluate-skill` child per included skill (at most 25), each
spending that skill's own budget. The scale guard asks before every run: a
`workflow()` child's agents are never counted.

**Skip the harness when:** the plugin has fewer than 2 eval-ready skills — the
modal case today, and `FLOOR = 2` is a hard bound that aborts there rather than
a tunable knob, because a single skill is exactly what `/evaluate:skill` already
does without an inventory agent, a nested workflow and an aggregate agent on
top. The steps below remain the authoritative description of *what* each stage
must produce; the harness only fixes *how* the work is split.

`context: fork` stays **off** for this skill. The fan-out here is a caller-chosen
`--parallel N` width, so the `[1m]` concurrent-subagent cascade hazard in
[`.claude/rules/skill-fork-context.md`](../../../.claude/rules/skill-fork-context.md)
applies; `scripts/plugin-compliance-check.sh` keeps this skill out of its
`context: fork` pin list for that reason, and the harness does not change it.

## Execution

### Step 1: Discover skills

Find all skills in the plugin:
```
<plugin-name>/skills/*/SKILL.md
```

List them and count the total.

### Step 2: Filter and prepare

For each skill, check if `evals.json` exists:
- **Has evals**: include in evaluation
- **No evals + `--create-missing-evals`**: include, will create evals during evaluation
- **No evals, no flag**: skip with a note

Report the breakdown:
```
Found N skills in <plugin-name>:
  - M with eval cases
  - K without eval cases (skipped | will create)
```

### Step 3: Run evaluations

For each included skill, invoke `/evaluate:skill` via the SlashCommand tool:

```
SlashCommand: /evaluate:skill <plugin-name>/<skill-name> [--create-evals] [--harness <h>]
```

Forward `--harness` unchanged to every skill when it was given; omit it otherwise
(the default is `subagent`).

If `--parallel N` is set and N > 1, batch evaluations into groups of N. Otherwise, run sequentially.

Report progress from the results themselves: after each `/evaluate:skill`
returns, state which skill finished and its pass rate from `benchmark.json`.
(Todo/task tools are not available on Opus 4.8+/Sonnet 5/Fable-generation
models unless `CLAUDE_CODE_ENABLE_TODO_TOOLS=1` is set; do not depend on them
— see `.claude/rules/agentic-permissions.md` § Task-tool availability.)

### Step 4: Aggregate plugin report

After all skill evaluations complete, read each skill's `benchmark.json` and aggregate:

```
bash evaluate-plugin/scripts/aggregate_benchmark.sh <plugin-name>
```

Write aggregated results to `<plugin-name>/eval-results/plugin-benchmark.json`.

### Step 5: Report

Print a plugin-level summary table:

```
## Plugin Evaluation: <plugin-name>

| Skill | Evals | Pass Rate | Status |
|-------|-------|-----------|--------|
| skill-a | 4 | 100% | PASS |
| skill-b | 3 | 67% | PARTIAL |
| skill-c | 5 | 80% | PASS |

**Overall**: 82% pass rate across N eval cases
```

Rank skills by pass rate. Flag any below 50% as needing attention.

## Agentic Optimizations

| Context | Command |
|---------|---------|
| Inventory plugin skills + evals | `bash evaluate-plugin/scripts/inspect_eval.sh --plugin-dir <plugin>` |
| Inspect a single skill's evals | `bash evaluate-plugin/scripts/inspect_eval.sh --plugin <plugin> --skill <skill>` |
| Aggregate results | `bash evaluate-plugin/scripts/aggregate_benchmark.sh <plugin>` |

## Quick Reference

| Flag | Description |
|------|-------------|
| `--create-missing-evals` | Generate eval cases for skills without them |
| `--parallel N` | Max concurrent evaluations (default: 1) |
| `--harness headless` | Run every skill's rollouts through real `claude -p` children (default: `subagent`) |
Files in this package (2)
  • SKILL.md 8 KB
  • workflows/evaluate-plugin-batch.workflow.js 16 KB

Versions

The database has no other skills named evaluate-plugin-batch.

Database snapshot: 13.11.2026 · package version deef6b4d00c5 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100evaluate-plugin
    Evaluate plugin quality. Use when user says \"evaluate plugin\", \"review plugin quality\", \"score my plugin\", \"check plugin\", \"rate plugin\".
  • 100check
    Run every available check against a Claude Code plugin — the first-party validator, plugin-dev's linters, and three checks nothing else performs. Use when asked to validate, check, audit, or review a plugin, or before publishing one.
  • 100local-evaluate-plugin-cf170bc7
    Evaluate a local Codex plugin in engineer-friendly language. Use when the user says "evaluate this plugin", "audit this plugin", "why did this score that way", "what should I fix first", "help me benchmark this plugin", or asks for a plugin-wide report before comparing versions.
  • 100plugin-review
    Review plugin quality with tiered checks and dependency scoping. Use for PR and pre-release audits.
  • 100plugin-builder
    Use when hardening, converting, auditing, or pre-release checking a Codex plugin package by verifying manifest paths, bundled skills, hooks, MCP/app config, validation gates, and release blockers.
  • 95evaluate-report
    View evaluation results and benchmark reports for a skill or plugin. Use when reviewing past eval results, comparing benchmark runs, or tracking quality trends.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.