Skip to content
Skills Library
AI & agents · Testing

add-eval

/add-eval

Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when evals:compare reports a drop.

Licence
MIT
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when evals:compare reports a drop.

Original author description

Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when evals:compare reports a drop.

How it works

  1. Layout
  2. Write a case
  3. Grader rules learned the hard way
  4. Mocks
  5. Run and compare
Original SKILL.md headings · 64 lines, 828 words
SKILL.md preview
---
name: add-eval
description: Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when evals:compare reports a drop.
---

Evals run the plugin in a sandboxed `claude -p` child against a mocked `platform` server and grade the transcript. Reference: https://code.claude.com/docs/en/plugin-evals.md (`claude plugin eval --help` matches it).

## Layout

```text
plugin/evals/
  <group>/<case>/prompt.md        frontmatter (tags, max_turns, allowed_tools) + the prompt, sent verbatim
  <group>/<case>/graders/<name>.md  one grader per file
  <group>/<case>/mocks/platform/<tool>.md  per-case override of one tool
  mocks/platform/_server.md       judge model plays a fixed demo tenant for every tool
  mocks/platform/_tools.json      real descriptions/schemas — generated, never hand-edit
  baseline.json                   committed per-grader pass rates, see "Run and compare"
  results/                        gitignored run output
```

Groups today: `skill-triggers/`, `tool-choice/`. The demo tenant (Demo Project, Billing Bot, Phone Support, GPT-4o) is defined in `_server.md`; write prompts against it.

## Write a case

1. Copy the closest existing case. The prompt is phrased like a user, never names the skill or tool, and contains everything needed — the run starts in an empty directory and cannot ask follow-ups. Add "I'm not around to answer questions, so pick sensible defaults and go ahead." unless asking IS the behaviour under test (`tool-choice/backup-gate`).
2. Give it one grader on the steps (`tool_used` / `tool_order` / trace `regex`) and, where there is a result to judge, one on the outcome (`regex` on `last_message`, or a short `llm` rubric written as PASS/FAIL conditions).
3. Set `max_turns` generously (25–30 for build flows); hitting it is a run error. Judge-played mocks get 4 × `max_turns` calls.
4. Smoke it: `npm run evals -- --case <name> --runs 1 --keep-temp`, then read `<kept>/out/trace.jsonl` and `out/mock-calls.jsonl` to confirm the run took the path you meant — a pass can be for the wrong reason.
5. Run it at 3 runs and re-baseline (below).

MCP tools are named `mcp__plugin_cognigy_platform__<tool>`; plugin skills are invoked as `cognigy:<skill>`.

## Grader rules learned the hard way

- **Accept the subagent path.** The `Agent` tool is available in every run even when `allowed_tools` omits it, and models delegate to `cognigy-agent-builder` / `cognigy-voice-go-live`. A `tool_used: Skill` grader then fails although the plugin did the right thing. Use a trace regex that accepts either path, and make sure the subagent preloads the skill (`skills:` in its frontmatter):
  ```yaml
  type: regex
  target: trace
  pattern: '"skill":"(?:cognigy:)?agent-creation"|"subagent_type":"(?:cognigy:)?cognigy-agent-builder"'
  ```
- **Match call inputs, never bare names.** The trace's first line lists every agent, skill and tool name, so `pattern: cognigy-voice-go-live` always passes. Anchor on `"skill":"`, `"subagent_type":"` or `"name":"mcp__…"`.
- **`tool_used` counts calls made inside subagents** — no special handling needed for MCP calls.
- **`tool_order` compares the FIRST matching call of each side.** It cannot express "the retry after the decline succeeded" when the first attempt is held. Grade the mock's answer instead: `target: mock_calls`, `pattern: '\\?"changesApplied\\?"\s*:\s*true'`, `arm: both`.
- **"Must not happen" graders** (`min: 0, max: 0`) pass for free in the no-plugin arm. Fine for `npm run evals` (single arm); just don't read them as signal there.
- No `(?i)` in patterns — use `flags: i`. `expect:` in mocks takes a restricted regex dialect (no groups/alternation; use a list of literals).

## Mocks

- `_server.md` answers every tool from the tenant description. To make one tool behave differently for one case, add `<case>/mocks/platform/<tool>.md` — it overrides only that tool.
- Fixed mock: body is the literal tool result (copy real shapes, incl. `_hints`, from `src/tools/handlers.ts`). Stateful behaviour: `type: agent` with instructions — agent mocks see earlier calls to every mocked tool, which is how `voice-apply-fixes` holds the first `apply: true` until `manage_snapshots decline`.
- After changing `src/tools/definitions.ts`: `npm run evals:tools`. A new tool must also be added to `tools:` in `_server.md`. `evalMocks.test.ts` fails on either drift.
- Mocks carry no `src/instructions.ts`, so evals measure skills, agents and tool descriptions — not the server instructions.
- `plugin/evals/` is in `.prettierignore` on purpose: Prettier escapes `_` in markdown and corrupts JSON mock bodies. Don't format these files.

## Run and compare

```bash
npm run evals                          # full suite, pinned Sonnet agent + Haiku judge
npm run evals -- --case 'voice-*'      # subset (globs on case name), --tag works too
npm run evals:compare                  # newest run vs plugin/evals/baseline.json; exit 1 on a drop
npm run evals:compare -- --update      # newest run becomes the baseline
```

- A drop with 3 runs (3/3 → 2/3) can be one flaky run. Re-run that case with `--runs 5`, and on `main` if still unsure, before calling it a regression.
- `changed` means the case's files (or shared mocks) differ from the baseline's fingerprint — expected after editing a case; re-baseline.
- `ERRORED` means a run ended abnormally (timeout, usage or rate limit); errored runs are still graded, so they are never compared — re-run. `--update` refuses a run that errored or skipped any case, so re-baseline from a clean full run, in the PR that intentionally changes behaviour or a case, so the `baseline.json` diff is reviewed. Commit the code first so the baseline records the commit it ran on. If you change the pinned models in `package.json`, re-baseline too.
- Never commit `results/`. Runs make paid model calls on your own credential; `--max-cost-usd` caps a run.

Versions

RepositoryScoreLicence
This page · cognigy/cognigy-plugin100MIT
umbraco/Umbraco-MCP-Base100MIT

Database snapshot: 13.11.2026 · package version 3b01220ecdda (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 86write-eval-case
    Authors one eval case in the Claude Code plugin eval format: a prompt a stranger can run, a scaffold script for fixtures, deterministic graders (regex, tool_used, tool_order, file_exists) before any llm grader, and the measured dimension in the tags. Use when asked to add a test…
  • 100harness-eval
    Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or…
  • 100skill-eval
    Measure whether a skill actually helps: build a `claude plugin eval` suite, run it with and without the skill under a required spend cap, and read the delta. Use when adding evals to a skill, converting evals.json, re-checking skills after a model upgrade, or deciding if a skill…
  • 90write-case
    Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the…
  • 84validate
    Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample…
  • 84plugin-eval
    Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, and reading the with-versus-without delta. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.