Skip to content
Skills Library
AI & agents · Testing

write-case

/write-case

Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the hook fired". Do not use to run the suite or explain a red result (that is run), or to fix the setup after a regression (that is repair).

Licence
Apache-2.0
Last repository change
27.10.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the hook fired". Do not use to run the suite or explain a red result (that is run), or to fix the setup after a regression (that is repair).

Triggers from the description

write an eval foradd a test case for this skill/hookmy grader is wronghow do I assert the hook fired
Original author description

Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the hook fired". Do not use to run the suite or explain a red result (that is run), or to fix the setup after a regression (that is repair).

How it works

  1. Rules that came from real failures
Original SKILL.md headings · 65 lines, 710 words
SKILL.md preview
---
name: write-case
description: Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the hook fired". Do not use to run the suite or explain a red result (that is run), or to fix the setup after a regression (that is repair).
---

# config-drift-checker: write-case

Layout: `<plugin>/evals/<case-dir>/prompt.md`, `graders/<name>.md`, optional `case.yaml`.

**prompt.md** frontmatter: `name`, `description` (one sentence a reviewer would read in the report:
what this case proves and which part of the setup it exercises), `tags: [..]`, a `covers.yaml` sidecar next to prompt.md (the
rule ids this case exercises; `node ${CLAUDE_PLUGIN_ROOT}/tools/config-coverage.mjs <plugin> --list`
prints them; a negative-trigger case covers nothing, that is correct). Target ids the `--list`
output marks `·` (uncovered) before adding a second case for a `✓` one. Every new case should move
the coverage %, which the Action reports and can enforce with `coverage-min`. `runs` (3), `max_turns`,
`timeout_seconds`, `allowed_tools: [Bash]` (only what the case needs), `model`. Body = the user
prompt, written the way a real user would write it (do not mention the skill or hook by name).

**graders/*.md** frontmatter `type`:
- `regex`: `pattern`, `flags`, `match: contains|not_contains|count:N`, `target`
- `tool_used`: `tool`, `input_match`, `min`, `max`
- `file_exists`: `path` (glob), `exists`
- `llm`: `focus` (never `target`: the official runner rejects it on llm graders); the file body is
  the criteria (one paragraph; "score 1 only if all hold")
- any grader: `weight`, and `arm: with-only` or `arm: both` (the only two values the official
  runner accepts)

What `target` (regex) and `focus` (llm) can point at, exactly as `claude plugin eval` defines it:
`last_message` (default), `trace`, `files` (the list of paths Claude *created*, not their
contents), or `{ source: file, path: <path> }` (the contents of one workspace file after the run).
To grade code, either ask the agent to print it ("Print the complete Java source in your reply")
and grade `last_message`, or grade the file it must write with `{ source: file, path: ... }`.
Body = one sentence a reviewer would understand.

**case.yaml** (`schema_version: "1.1"`, and `name` equal to the case's directory name so `--case`
globs match the folder under both runners): `context.scaffold_script` names a **script file** in the
case directory (`scaffold.sh`), never an inline script: it gives the agent a state to act on (a
git repo, a file, a branch). The official runner runs it with no `EVAL_*` variables, so a script
that needs the plugin's files locates itself:
`ROOT="${EVAL_PLUGIN_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"`.
`context.add_dirs` lists read-only fixture directories.

## Rules that came from real failures
1. `not_contains` graders must match **code position** (`^\s*@Autowired`, flag `m`), never a
   word that can appear in prose, because the model will say "I avoided X".
2. A negative-trigger case (`tool_used: Skill, max: 0, arm: both`) for every skill.
3. Hook cases: choose a command the model will *attempt* in a scratch repo (`git reset --hard
   HEAD`, `git clean -fd`), not one it refuses on its own (`git push --force origin main`).
   Assert three things: attempted (tool_used Bash), reported blocked (regex), not succeeded
   (regex not_contains on success phrases like "HEAD is now at|ran successfully").
4. `tool_used: Skill` graders are indicators under ablation, not score. That is correct; add
   `arm: both` only when the assertion must hold without the plugin too.
5. Keep prompts short and realistic; put setup in `scaffold_script`, not in the prompt.
6. Regex over Java/TS: never rely on `name\([^)]*\)` to find a method; parameter annotations
   contain parentheses. Anchor on something unique and flat (the mapping annotation string, the
   record name) and use a bounded lazy window `[\s\S]{0,900}?`.
7. Test every regex against a hand-written snippet of the expected code with `node -e` **before**
   spending an agent run; a grader that fails a correct answer costs more trust than it saves.
8. **Safety (learned the hard way):** the without-arm has no guard, and workspaces do not
   sandbox Docker, the network, or the host. A hook case's command must be harmless when it
   *succeeds*: scratch git repo, or a stub binary created by `scaffold_script` in `.eval-bin/`
   (`printf '#!/usr/bin/env bash\necho stub $*\n' > .eval-bin/docker-compose; chmod +x …`).
   The runner's safety net blocks compose down -v / prune / force-push / rm -rf outside the
   workspace unless the binary is stubbed. Check for `name:`/`container_name:` in compose files
   and for real remotes before writing the prompt.
9. Turn budgets: real-code cases need `max_turns: 30` (read → edit 2–4 files → test); single-file
   or prose cases 12; hook cases 6. Runs that hit the cap are scored as-is and flagged TRUNCATED;
   a signal to raise the budget, not a failure of the setup.

Versions

The database has no other skills named write-case.

Database snapshot: 13.11.2026 · package version 51b895fb56fc (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 86write-eval-case
    Authors one eval case in the Claude Code plugin eval format: a prompt a stranger can run, a scaffold script for fixtures, deterministic graders (regex, tool_used, tool_order, file_exists) before any llm grader, and the measured dimension in the tags. Use when asked to add a test…
  • 100harness-eval
    Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or…
  • 84validate
    Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample…
  • 100add-eval
    Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when…
  • 100repair
    After a red eval run, propose the smallest change to the agent setup (CLAUDE.md, a skill, a hook) that restores the drifted behaviour, verify it by re-running only the failing cases, and write a PR-ready summary. Use when the user says \"fix the regression\", \"repair the…
  • 100run
    Run the agent-config eval suite locally and compare with the stored baseline; explain why a case went red. Use when the user says "run the evals", "run config-drift-checker", "did the Claude Code update break my setup", "why is the eval red", or "compare against baseline". Do…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.