Skip to content
Skills Library
AI & agents · Testing

skill-eval

/skill-eval

Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval

Licence
MIT
Last repository change
15.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
9
Scripts
3
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 15.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 3 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/__init__.py, scripts/aggregate_benchmark.py, scripts/run_eval.py

What it does and when to use it

Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval

Original author description

Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval

How it works

  1. Setup
  2. Run Evals
  3. Draft Assertions
  4. Grade
  5. Aggregate
  6. Analyze Benchmark
  7. Blind Comparison
  8. Post-Hoc Analysis
  9. Report
  10. Iterating
Original SKILL.md headings · 115 lines, 466 words
SKILL.md preview
---
name: skill-eval
description: Use when evaluating skill output quality, comparing skill versions, running behavioral evals, benchmarking a skill, or when triggered by /skill-eval
---

# Skill Evaluation

Evaluate skill output quality with assertion-based grading, blind before/after comparison, and variance analysis across 3 runs per scenario.

## Setup

1. **Resolve paths** — run once at the start of any eval session:

   ```bash
   PLUGIN_ROOT=$(python3 -c "
   import json, os
   p = json.load(open(os.path.expanduser('~/.claude/plugins/installed_plugins.json')))
   print(p['plugins']['claude-caliper@claude-caliper'][0]['installPath'])
   ")
   EVAL_ROOT=~/.claude/skill-evals
   SKILL_EVAL_SCRIPT=$PLUGIN_ROOT/skills/skill-eval/scripts/run_eval.py
   AGGREGATE_SCRIPT=$PLUGIN_ROOT/skills/skill-eval/scripts/aggregate_benchmark.py
   ```

2. **Identify target** and what "before" represents:
   - New skill: before = no skill (`type: "none"`)
   - Improvement: before = snapshot of previous version
   - Snapshot before editing: `cp -r skills/{name}/SKILL.md $EVAL_ROOT/{name}/snapshot-before/SKILL.md`

3. **Check for evals.json** at `$EVAL_ROOT/{name}/evals.json`. If none, create interactively — 2-3 realistic prompts including adversarial scenarios (deadline pressure, "skip this step", ambiguous requirements). Use the schema in `references/schemas.md`: top-level object with `skill_name` (string) and `evals` (array). Each eval needs integer `id`, `name`, `prompt`, and `expectations`.

4. **Write config.json** for this iteration at `$EVAL_ROOT/{name}/iteration-N/config.json`:
   ```json
   {
     "before": {"label": "v1", "type": "skill", "skill_path": "~/.claude/skill-evals/{name}/snapshot-before/SKILL.md"},
     "after": {"label": "v2", "type": "skill", "skill_path": "skills/{name}/SKILL.md"}
   }
   ```

**See:** references/schemas.md for all JSON formats.

## Run Evals

Spawn both variants simultaneously — don't run before first then after:

```bash
# After variant
python3 $SKILL_EVAL_SCRIPT \
  --evals-path $EVAL_ROOT/{name}/evals.json \
  --output-dir $EVAL_ROOT/{name}/iteration-N/ \
  --variant after \
  --skill-path skills/{name}/SKILL.md \
  --runs 3

# Before variant
python3 $SKILL_EVAL_SCRIPT \
  --evals-path $EVAL_ROOT/{name}/evals.json \
  --output-dir $EVAL_ROOT/{name}/iteration-N/ \
  --variant before \
  --skill-path $EVAL_ROOT/{name}/snapshot-before/SKILL.md \
  --runs 3
```

For no-skill baseline, omit `--skill-path`.

Each eval creates: `$EVAL_ROOT/{name}/iteration-N/eval-{id}-{slug}/{variant}/run-{n}/output.txt` and `timing.json`.

## Draft Assertions

While runs execute, draft or refine assertions in evals.json. Good assertions are objectively verifiable, descriptively named, and test behavioral outcomes (not surface compliance). The grader flags trivially-satisfied assertions.

## Grade

**See:** agents/grader.md

Spawn a grader subagent per eval. For each run, the grader reads `output.txt`, evaluates each assertion with cited evidence, self-critiques assertion quality, and writes `grading.json` per run directory.

PASS requires genuine task completion with cited evidence, not keyword matching.

## Aggregate

```bash
python3 $AGGREGATE_SCRIPT \
  $EVAL_ROOT/{name}/iteration-N \
  --skill-name {name}
```

Produces `benchmark.json` (pass_rate, time with mean +/- stddev, delta) and `benchmark.md`.

## Analyze Benchmark

**See:** agents/analyzer.md (Mode 2: Benchmark Pattern Analysis)

Spawn analyzer subagent to surface patterns aggregates hide: non-discriminating assertions (always pass), high-variance evals (flaky), time tradeoffs.

## Blind Comparison

**See:** agents/comparator.md

Spawn comparator subagent with representative outputs from before and after WITHOUT revealing which is which. Scores Content (1-5) + Structure (1-5) -> Overall (1-10). Saves `comparison.json`.

## Post-Hoc Analysis

**See:** agents/analyzer.md (Mode 1: Post-Hoc Comparison Analysis)

Spawn analyzer with comparison results + both SKILL.md files. Unblinds results, identifies why winner won, scores instruction-following (1-10), generates prioritized improvement suggestions. Saves `analysis.json`.

## Report

Present to user:
- Pass rates per variant with variance (mean +/- stddev)
- Blind comparison winner + reasoning
- Improvement suggestions (prioritized High/Medium/Low)
- Assertion quality feedback (strengthen/drop recommendations)

## Iterating

Edit skill -> snapshot as new "before" -> rerun into `iteration-N+1/` -> repeat until satisfied.
Files in this package (9)
  • README.md 10 KB
  • SKILL.md 4 KB
  • agents/analyzer.md 5 KB
  • agents/comparator.md 4 KB
  • agents/grader.md 4 KB
  • references/schemas.md 8 KB
  • scripts/__init__.py 1 KB
  • scripts/aggregate_benchmark.py 14 KB
  • scripts/run_eval.py 8 KB

Versions

RepositoryScoreLicence
This page · nikhilsitaram/claude-caliper100MIT
boshu2/agentops100Apache-2.0
joris887/exosuit100MIT
lamngockhuong/aiteamkit100MIT
ivy00johns/skill-madness100MIT
The-Utopia-Studio/skills100Apache-2.0
lingegrace/claude-research-skills100MIT
PIXARTSeu/Synapse100AGPL-3.0

Database snapshot: 13.11.2026 · package version 74357a72d35b (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100bmad-eval-runner
    Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs.
  • 100plugin-evaluation
    Measures whether a Claude Code plugin actually works by running triggering evals (does the model pick the skill?) and output evals (does it produce correct results?). Use when you need to evaluate a plugin, run skill activation testing, set up an eval harness, measure plugin…
  • 100skill-eval-scaffolder
    Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution.
  • 100eval-prompt
    Design and run a small evaluation for a prompt, system prompt, Agent Skill or LLM feature, and compare versions. Use when the user asks "is this prompt better?", "test my skill", "evaluate the agent", "build an eval", or wants to change a prompt without breaking what works.
  • 95skill-benchmarking
    Run skill benchmarks with discriminating-only assertions against evals.json for any model and any AI agent. Use when benchmarking a skill against a model not yet tested, running with_skill/without_skill eval pairs, producing benchmark-<model>.json, re-grading an existing run,…
  • 100run-evals
    Run skill evals with deterministic output layout and CI-compatible results. Grades assertions, aggregates benchmark.json, and renders summary.md via Python scripts. NOT skill-creator — this skill produces fixed output paths for automated pipelines. Triggers on "run evals", "eval…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.