Skip to content
Skills Library
AI & agents · Testing

init-plugin-eval

/init-plugin-eval

Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.

Licence
MIT
Last repository change
14.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 14.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.

Triggers from the description

set up evals for this plugininit plugin evalsscaffold eval fixturescreate the eval harness for <plugin>

Arguments

/init-plugin-eval <plugin-name> [--dir evals/<plugin-name>]
Original author description

Scaffold the eval directory structure for a plugin's review agents and advisory skills. Use after creating a plugin or adding its first review agent, or when the user says "set up evals for this plugin", "init plugin evals", "scaffold eval fixtures", or "create the eval harness for <plugin>". Creates the fixtures/ and expected/ dirs plus a README describing the grading contract.

How it works

  1. Constraints
  2. Step 1 — Parse
  3. Step 2 — Create the structure
  4. Step 3 — Write the grading-contract README
  5. Step 4 — Verify the empty scaffold is a clean no-op
  6. Completion report
Original SKILL.md headings · 81 lines, 429 words
SKILL.md preview
---
name: init-plugin-eval
description: >-
  Scaffold the eval directory structure for a plugin's review agents and advisory
  skills. Use after creating a plugin or adding its first review agent, or when
  the user says "set up evals for this plugin", "init plugin evals", "scaffold
  eval fixtures", or "create the eval harness for <plugin>". Creates the
  fixtures/ and expected/ dirs plus a README describing the grading contract.
argument-hint: "<plugin-name> [--dir evals/<plugin-name>]"
role: implementation
allowed-tools: Read, Write, Bash, Glob
---

# Init Plugin Eval

Role: implementation. Create the eval scaffold so a plugin's review agents and
advisory skills can be graded — without shipping any of it inside the plugin.

## Constraints

1. **Evals never ship.** The eval tree lives at the **repo root** under
   `evals/<plugin>/`, never under `plugins/<plugin>/`. Build/test tooling inside
   a plugin dir is a shipping bug.
2. **Stub, don't fake.** Scaffold empty `fixtures/` and `expected/` dirs and a
   README. Do **not** invent fixtures — an empty scaffold must grade as
   "no fixtures", a clean no-op, not an error.
3. **Never overwrite.** If `evals/<plugin>/` exists, report and stop.

## Step 1 — Parse

- `NAME` = `$0` (the plugin name, matching its `plugin.json` `name`).
- `DIR` = `--dir` value, else `evals/<NAME>`.
- If `DIR` exists → report and stop.

## Step 2 — Create the structure

```bash
mkdir -p "$DIR/fixtures" "$DIR/expected"
for d in fixtures expected; do : > "$DIR/$d/.gitkeep"; done
```

- `fixtures/` — input artifacts an agent/skill runs against (sample agent files,
  plugin dirs, prose use-cases).
- `expected/` — one `<fixture-stem>.json` per fixture declaring the expected
  grading (which agent/skill it targets and the pass condition).

## Step 3 — Write the grading-contract README

**`$DIR/README.md`** documents the contract a grader enforces:

```markdown
# Eval corpus — <NAME>

Fixtures (`fixtures/`) and their expected gradings (`expected/*.json`) for the
<NAME> plugin's review agents and advisory skills.

## Grading contract

Each `expected/<stem>.json` pairs with a `fixtures/<stem>.*` input and declares:

- `fixture` — the fixture file/dir this grades.
- `description` — what the fixture exercises.
- `applicableAgents` / `applicableSkills` — the unit(s) under test.
- For a **review agent**: `expectedStatus` (`pass|warn|fail|skip`),
  `issueCount` ({min,max}), and `severities` bounds.
- For an **advisory skill**: the expected `recommendation`/fields and the
  rule citations or gates the output must contain.

A grader pairs every `expected/*.json` with a fixture and checks the actual
output falls within the declared bounds. An **empty corpus grades as
"no fixtures"** — a clean no-op, never an error.
```

## Step 4 — Verify the empty scaffold is a clean no-op

Confirm the scaffold grades as "no fixtures", not an error:

- `expected/` contains only `.gitkeep` (no `*.json`) → a corpus check finds
  nothing to grade and reports "no fixtures".
- If a project-level eval runner exists (e.g. `scripts/eval_grade.py
  --check-corpus`), run it scoped to `evals/<NAME>/` and confirm exit 0 with a
  "no fixtures" / "nothing to grade" message.

## Completion report

```text
Eval scaffold created: <DIR>
Dirs:  fixtures/ expected/
Docs:  README.md (grading contract)
State: no fixtures yet → grades as a clean no-op
Next:  add a fixtures/<stem>.* input and an expected/<stem>.json grading
```

Versions

The database has no other skills named init-plugin-eval.

Database snapshot: 13.11.2026 · package version 913bbacf7cda (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 95agent-eval
    Run eval fixtures against review agents and grade results. Use this after adding or modifying a review agent, to validate detection accuracy, or when the user says "run the evals", "test the agents", "check for regressions", or "how accurate is the agent".
  • 86write-eval-case
    Authors one eval case in the Claude Code plugin eval format: a prompt a stranger can run, a scaffold script for fixtures, deterministic graders (regex, tool_used, tool_order, file_exists) before any llm grader, and the measured dimension in the tags. Use when asked to add a test…
  • 100add-eval
    Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when…
  • 84design
    Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite',…
  • 100skill-eval
    Measure whether a skill actually helps: build a `claude plugin eval` suite, run it with and without the skill under a required spend cap, and read the delta. Use when adding evals to a skill, converting evals.json, re-checking skills after a model upgrade, or deciding if a skill…
  • 100skill-eval-scaffolder
    Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.