skill-eval
/skill-eval
Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval
- Author
- nikhilsitaram ↗
- Repository
- claude-caliper ↗
- Licence
- MIT
- Last repository change
- 15.10.2026
Assessment and security
- Files in this package
- 9
- Scripts
- 3
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 15.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 3 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/__init__.py, scripts/aggregate_benchmark.py, scripts/run_eval.py
What it does and when to use it
Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval
Original author description
Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval
How it works
- Setup
- Run Evals
- Draft Assertions
- Grade
- Aggregate
- Analyze Benchmark
- Blind Comparison
- Post-Hoc Analysis
- Report
- Iterating
SKILL.md preview
---
name: skill-eval
description: Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval
---
# Skill Evaluation
Evaluate skill output quality with assertion-based grading, blind before/after comparison, and variance analysis across 3 runs per scenario.
## Setup
1. **Resolve paths** — run once at the start of any eval session:
```bash
PLUGIN_ROOT=$(python3 -c "
import json, os
p = json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json')))
print(p['plugins']['claude-caliper@claude-caliper'][0]['installPath'])
")
EVAL_ROOT=~/.claude/skill-evals
SKILL_EVAL_SCRIPT=$PLUGIN_ROOT/skills/skill-eval/scripts/run_eval.py
AGGREGATE_SCRIPT=$PLUGIN_ROOT/skills/skill-eval/scripts/aggregate_benchmark.py
```
2. **Identify target** and what "before" represents:
- New skill: before = no skill (`type: "none"`)
- Improvement: before = snapshot of previous version
- Snapshot before editing: `cp -r skills/{name}/SKILL.md $EVAL_ROOT/{name}/snapshot-before/SKILL.md`
3. **Check for evals.json** at `$EVAL_ROOT/{name}/evals.json`. If none, create interactively — 2-3 realistic prompts including adversarial scenarios (deadline pressure, "skip this step", ambiguous requirements). Use the schema in `references/schemas.md`: top-level object with `skill_name` (string) and `evals` (array). Each eval needs integer `id`, `name`, `prompt`, and `expectations`.
4. **Write config.json** for this iteration at `$EVAL_ROOT/{name}/iteration-N/config.json`:
```json
{
"before": {"label": "v1", "type": "skill", "skill_path": "~/.claude/skill-evals/{name}/snapshot-before/SKILL.md"},
"after": {"label": "v2", "type": "skill", "skill_path": "skills/{name}/SKILL.md"}
}
```
**See:** references/schemas.md for all JSON formats.
## Run Evals
Spawn both variants simultaneously — don't run before first then after:
```bash
# After variant
python3 $SKILL_EVAL_SCRIPT \
--evals-path $EVAL_ROOT/{name}/evals.json \
--output-dir $EVAL_ROOT/{name}/iteration-N/ \
--variant after \
--skill-path skills/{name}/SKILL.md \
--runs 3
# Before variant
python3 $SKILL_EVAL_SCRIPT \
--evals-path $EVAL_ROOT/{name}/evals.json \
--output-dir $EVAL_ROOT/{name}/iteration-N/ \
--variant before \
--skill-path $EVAL_ROOT/{name}/snapshot-before/SKILL.md \
--runs 3
```
For no-skill baseline, omit `--skill-path`.
Each eval creates: `$EVAL_ROOT/{name}/iteration-N/eval-{id}-{slug}/{variant}/run-{n}/output.txt` and `timing.json`.
## Draft Assertions
While runs execute, draft or refine assertions in evals.json. Good assertions are objectively verifiable, descriptively named, and test behavioral outcomes (not surface compliance). The grader flags trivially-satisfied assertions.
## Grade
**See:** agents/grader.md
Spawn a grader subagent per eval. For each run, the grader reads `output.txt`, evaluates each assertion with cited evidence, self-critiques assertion quality, and writes `grading.json` per run directory.
PASS requires genuine task completion with cited evidence, not keyword matching.
## Aggregate
```bash
python3 $AGGREGATE_SCRIPT \
$EVAL_ROOT/{name}/iteration-N \
--skill-name {name}
```
Produces `benchmark.json` (pass_rate, time with mean +/- stddev, delta) and `benchmark.md`.
## Analyze Benchmark
**See:** agents/analyzer.md (Mode 2: Benchmark Pattern Analysis)
Spawn analyzer subagent to surface patterns aggregates hide: non-discriminating assertions (always pass), high-variance evals (flaky), time tradeoffs.
## Blind Comparison
**See:** agents/comparator.md
Spawn comparator subagent with representative outputs from before and after WITHOUT revealing which is which. Scores Content (1-5) + Structure (1-5) -> Overall (1-10). Saves `comparison.json`.
## Post-Hoc Analysis
**See:** agents/analyzer.md (Mode 1: Post-Hoc Comparison Analysis)
Spawn analyzer with comparison results + both SKILL.md files. Unblinds results, identifies why winner won, scores instruction-following (1-10), generates prioritized improvement suggestions. Saves `analysis.json`.
## Report
Present to user:
- Pass rates per variant with variance (mean +/- stddev)
- Blind comparison winner + reasoning
- Improvement suggestions (prioritized High/Medium/Low)
- Assertion quality feedback (strengthen/drop recommendations)
## Iterating
Edit skill -> snapshot as new "before" -> rerun into `iteration-N+1/` -> repeat until satisfied.
Files in this package (9)
- README.md 10 KB
- SKILL.md 4 KB
- agents/analyzer.md 5 KB
- agents/comparator.md 4 KB
- agents/grader.md 4 KB
- references/schemas.md 8 KB
- scripts/__init__.py 1 KB
- scripts/aggregate_benchmark.py 14 KB
- scripts/run_eval.py 8 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · nikhilsitaram/claude-caliper | 100 | MIT |
| boshu2/agentops | 100 | Apache-2.0 |
| joris887/exosuit | 100 | MIT |
| lamngockhuong/aiteamkit | 100 | MIT |
| ivy00johns/skill-madness | 100 | MIT |
| The-Utopia-Studio/skills | 100 | Apache-2.0 |
| lingegrace/claude-research-skills | 100 | MIT |
| PIXARTSeu/Synapse | 100 | AGPL-3.0 |
Database snapshot: 13.11.2026 · package version 74357a72d35b (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100bmad-eval-runner
Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs. - 100plugin-evaluation
Measures whether a Claude Code plugin actually works by running triggering evals (does the model pick the skill?) and output evals (does it produce correct results?). Use when you need to evaluate a plugin, run skill activation testing, set up an eval harness, measure plugin… - 100skill-eval-scaffolder
Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution. - 100eval-prompt
Design and run a small evaluation for a prompt, system prompt, Agent Skill or LLM feature, and compare versions. Use when the user asks "is this prompt better?", "test my skill", "evaluate the agent", "build an eval", or wants to change a prompt without breaking what works. - 95skill-benchmarking
Run skill benchmarks with discriminating-only assertions against evals.json for any model and any AI agent. Use when benchmarking a skill against a model not yet tested, running with_skill/without_skill eval pairs, producing benchmark-<model>.json, re-grading an existing run,… - 100run-evals
Run skill evals with deterministic output layout and CI-compatible results. Grades assertions, aggregates benchmark.json, and renders summary.md via Python scripts. NOT skill-creator — this skill produces fixed output paths for automated pipelines. Triggers on "run evals", "eval…