init-plugin-eval
/init-plugin-eval
Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.
- Author
- bdfinst ↗
- Repository
- agentic-dev-team ↗
- Licence
- MIT
- Last repository change
- 14.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 14.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.
Triggers from the description
Arguments
/init-plugin-eval <plugin-name> [--dir evals/<plugin-name>]
Original author description
Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.
How it works
- Constraints
- Step 1 — Parse
- Step 2 — Create the structure
- Step 3 — Write the grading-contract README
- Step 4 — Verify the empty scaffold is a clean no-op
- Completion report
SKILL.md preview
---
name: init-plugin-eval
description: >-
Scaffold the eval directory structure for a plugin's review agents and advisory
skills. Use after creating a plugin or adding its first review agent, or when
the user says "set up evals for this plugin", "init plugin evals", "scaffold
eval fixtures", or "create the eval harness for <plugin>". Creates the
fixtures/ and expected/ dirs plus a README describing the grading contract.
argument-hint: "<plugin-name> [--dir evals/<plugin-name>]"
role: implementation
allowed-tools: Read, Write, Bash, Glob
---
# Init Plugin Eval
Role: implementation. Create the eval scaffold so a plugin's review agents and
advisory skills can be graded — without shipping any of it inside the plugin.
## Constraints
1. **Evals never ship.** The eval tree lives at the **repo root** under
`evals/<plugin>/`, never under `plugins/<plugin>/`. Build/test tooling inside
a plugin dir is a shipping bug.
2. **Stub, don't fake.** Scaffold empty `fixtures/` and `expected/` dirs and a
README. Do **not** invent fixtures — an empty scaffold must grade as
"no fixtures", a clean no-op, not an error.
3. **Never overwrite.** If `evals/<plugin>/` exists, report and stop.
## Step 1 — Parse
- `NAME` = `$0` (the plugin name, matching its `plugin.json` `name`).
- `DIR` = `--dir` value, else `evals/<NAME>`.
- If `DIR` exists → report and stop.
## Step 2 — Create the structure
```bash
mkdir -p "$DIR/fixtures" "$DIR/expected"
for d in fixtures expected; do : > "$DIR/$d/.gitkeep"; done
```
- `fixtures/` — input artifacts an agent/skill runs against (sample agent files,
plugin dirs, prose use-cases).
- `expected/` — one `<fixture-stem>.json` per fixture declaring the expected
grading (which agent/skill it targets and the pass condition).
## Step 3 — Write the grading-contract README
**`$DIR/README.md`** documents the contract a grader enforces:
```markdown
# Eval corpus — <NAME>
Fixtures (`fixtures/`) and their expected gradings (`expected/*.json`) for the
<NAME> plugin's review agents and advisory skills.
## Grading contract
Each `expected/<stem>.json` pairs with a `fixtures/<stem>.*` input and declares:
- `fixture` — the fixture file/dir this grades.
- `description` — what the fixture exercises.
- `applicableAgents` / `applicableSkills` — the unit(s) under test.
- For a **review agent**: `expectedStatus` (`pass|warn|fail|skip`),
`issueCount` ({min,max}), and `severities` bounds.
- For an **advisory skill**: the expected `recommendation`/fields and the
rule citations or gates the output must contain.
A grader pairs every `expected/*.json` with a fixture and checks the actual
output falls within the declared bounds. An **empty corpus grades as
"no fixtures"** — a clean no-op, never an error.
```
## Step 4 — Verify the empty scaffold is a clean no-op
Confirm the scaffold grades as "no fixtures", not an error:
- `expected/` contains only `.gitkeep` (no `*.json`) → a corpus check finds
nothing to grade and reports "no fixtures".
- If a project-level eval runner exists (e.g. `scripts/eval_grade.py
--check-corpus`), run it scoped to `evals/<NAME>/` and confirm exit 0 with a
"no fixtures" / "nothing to grade" message.
## Completion report
```text
Eval scaffold created: <DIR>
Dirs: fixtures/ expected/
Docs: README.md (grading contract)
State: no fixtures yet → grades as a clean no-op
Next: add a fixtures/<stem>.* input and an expected/<stem>.json grading
```
Versions
The database has no other skills named init-plugin-eval.
Database snapshot: 13.11.2026 · package version 913bbacf7cda (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 95agent-eval
Run eval fixtures against review agents and grade results. Use this after adding or modifying a review agent, to validate detection accuracy, or when the user says "run the evals", "test the agents", "check for regressions", or "how accurate is the agent". - 86write-eval-case
Authors one eval case in the Claude Code plugin eval format: a prompt a stranger can run, a scaffold script for fixtures, deterministic graders (regex, tool_used, tool_order, file_exists) before any llm grader, and the measured dimension in the tags. Use when asked to add a test… - 100add-eval
Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when… - 84design
Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite',… - 100skill-eval
Measure whether a skill actually helps: build a `claude plugin eval` suite, run it with and without the skill under a required spend cap, and read the delta. Use when adding evals to a skill, converting evals.json, re-checking skills after a model upgrade, or deciding if a skill… - 100skill-eval-scaffolder
Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution.