validate
/validate
Waliduje statycznie zestaw claude plugin eval (prompt.md, case.yaml, graders/*.md) przed uruchomieniem kosztownych przebiegów: błędy ładowania, ostrzeżenia o typowych pomyłkach i offline sprawdzenie graderów.
- Autor
- melodic-software ↗
- Repozytorium
- claude-code-plugins ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 26.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 5
- Skrypty
- 3
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 3
Brak sygnałów. Repozytorium zmienione 26.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
- 3 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.
Skrypty: scripts/test_validate_cases.py, scripts/validate-cases.py, scripts/validate-cases.test.sh
Co robi i kiedy
Waliduje statycznie zestaw claude plugin eval (prompt.md, case.yaml, graders/*.md) przed uruchomieniem kosztownych przebiegów: błędy ładowania, ostrzeżenia o typowych pomyłkach i offline sprawdzenie graderów.
Wyzwalacze z opisu
Argumenty
/validate [eval-dir]
Oryginalny opis (EN)
Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample answers. Use when: 'validate my eval cases', 'check my eval suite', 'will this suite load', 'lint case.yaml', 'check my graders', 'why did my case fail to load', or before paying for a run. Not for the skill-creator `evals/evals.json` format.
Jak działa
- Run it
- What the two tiers mean
- Sample answers
- When the parser stops
- Mirroring record
- Boundary
- Next
- Gotchas
Podgląd SKILL.md
---
description: "Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample answers. Use when: 'validate my eval cases', 'check my eval suite', 'will this suite load', 'lint case.yaml', 'check my graders', 'why did my case fail to load', or before paying for a run. Not for the skill-creator `evals/evals.json` format."
argument-hint: "[eval-dir]"
user-invocable: true
disable-model-invocation: false
metadata:
workflow-stage: test
summary: Static FAIL/WARN check of an eval suite's cases and graders, with no model call
---
# Validate eval cases before a run spends money
A case that fails to load costs the same as one that works. `scripts/validate-cases.py` reads the
suite off disk and reports what would break, with no model call and no network access.
## Run it
```bash
python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>
```
`<eval-dir>` is the directory holding the cases, `evals/` under the plugin root by default. Add
`--json` for the findings as an array of `{level, case, file, message}` objects; `--help` prints
the full contract from the script's header.
| Exit | Meaning |
|---|---|
| 0 | no FAIL finding. WARN findings are printed and do not gate |
| 1 | at least one FAIL finding |
| 2 | usage error, or the eval dir is missing or unreadable |
Findings print one per line as `<FAIL|WARN> <case>/<file>: <message>`, sorted by case then file; a
finding about the suite as a whole carries the eval dir in place of `<case>/<file>`.
## What the two tiers mean
**FAIL is what the binary itself rejects**: an unknown `prompt.md` frontmatter key, a grader with no
usable `type`, an unknown option for the declared grader type, a case with no grader, two graders
sharing a name, a non-positive `weight`, `runs` / `max_turns` / `timeout_seconds` out of bounds, an
`env` key that does not match `EVAL_[A-Z0-9_]*`, a case that has a `case.yaml` but no
`schema_version` or `name` in it or in `prompt.md` frontmatter (without a `case.yaml` the binary
supplies both), a case with no prompt (`prompt.md` body or `execution.prompt`), a grader missing an
option its type requires, and a `schema_version` that is not a quoted version string or whose
major is newer than the binary supports. The complete sets the script checks, so a question about
a key or a type is answered here without opening the script:
- `prompt.md` frontmatter keys: `schema_version`, `name`, `description`, `tags`, `plugins`,
`runs`, `expected_outcome`, `model`, `max_turns`, `timeout_seconds`, `allowed_tools`,
`append_system_prompt`, `env`. A time limit is `timeout_seconds`; `timeout` is an unknown key, so
the case fails to load.
- Grader `type`: `regex`, `tool_order`, `tool_used`, `file_exists`, `llm`, `baseline`. Any other
value, such as `contains` or `string_match`, is no usable type, so the case fails to load. A
substring or pattern check is `type: regex` with the text in `pattern`.
- Options per type, beside the `type`, `weight`, `arm` and `name` every grader takes: `regex`
`pattern`, `flags`, `match`, `target`; `tool_used` `tool`, `input_match`, `min`, `max`;
`tool_order` `before`, `after`; `file_exists` `path`, `exists`; `llm` `criteria`, `focus`;
`baseline` `baseline_file`, `criteria`.
- Required options per type: `regex` `pattern`; `tool_used` `tool`; `tool_order` `before`,
`after`; `file_exists` `path`; `llm` `criteria`; `baseline` `baseline_file`, `criteria`. A
`graders/*.md` body stands in for `pattern` (regex) or `criteria` (llm, baseline).
- Bounds: `runs` 1 to 50, `max_turns` 1 to 200, `timeout_seconds` 1 to 3600.
The script's constants are the one place to correct when the schema moves, under the drift record
its header carries; this list follows them.
**WARN is an authoring mistake the suite loads with** and then scores badly on: `target: files`
(which reads the list of paths created, not their contents), an inline `(?i)` the grader's regex
engine does not honor, a case whose graders are all judges (the two paid types, with no
deterministic grader beside them), a tool in `allowed_tools` that the operator must grant with
`--allow-tools`, `file_exists` in a case that requests no write tool (so nothing it could match is
ever created), and a `prompt.md` key the binary loads but the reference page does not list, which
can change without notice (the script's `UNDOCUMENTED_PROMPT_KEYS`, under its own drift record).
The judge-only rule is this skill's own pairing heuristic, not a rejection the binary makes; the
sample-answer findings below are likewise this skill's checks.
A WARN never sets exit 1, so a suite can ship with warnings on purpose. Read them once and decide.
## Sample answers
A grader that rejects a correct paraphrase scores the plugin down for nothing, and no run says the
grader was at fault. So each case can carry known answers, and the script runs the free graders
over them before any money is spent.
Put them in `samples/<grader-name>.json` inside the case directory, beside `graders/` and never in
it:
```json
{
"pass": [{"answer": "...", "why": "the ranking rule with 'and' for the comma"}],
"fail": [{"answer": ""}, {"answer": "...", "why": "near miss: base-model phrasing"}]
}
```
`answer` is what the grader reads: text for a `regex` (the final reply on the default target), a
list of `{"tool", "input"}` calls for `tool_used` and `tool_order`, and a list of created paths for
`file_exists`. `why` is optional and is echoed in any finding. Give each grader several paraphrases
it must pass, near misses it must reject, an empty answer, and an answer to a different question.
| Finding | Level |
|---|---|
| A must-pass answer the grader rejects, or a must-fail answer it accepts | FAIL |
| A sample file that is not JSON in this shape, or an answer of the wrong kind for its grader | FAIL |
| A `regex`, `tool_used`, `tool_order` or `file_exists` grader with no sample file | WARN |
| A sample file with no must-pass or no must-fail answers, or named after no grader | WARN |
| A setting the script cannot reproduce offline, such as the `y` or `v` regex flag | WARN |
| A pattern with Python-only syntax: `\A`, `\Z`, `\z`, a possessive quantifier, an atomic group, a `(?P<` group or `(?P=` reference, a `(?#` comment, a `(?(` conditional. The finding names the grader and the token | FAIL |
| With `node` on PATH: a regex that does not compile under JavaScript, or a sample JavaScript's `RegExp` sorts the other way | FAIL |
| An `llm` or `baseline` grader with samples: they need a paid judge calibration run | WARN |
The script FAILs the Python-only tokens listed above; that check needs nothing installed. When
`node` is on PATH it also grades each regex grader and its samples through `RegExp`, and without
`node` it skips that part silently.
The script never calls a judge. Samples on an `llm` grader are the labeled answers a calibration
run feeds the judge, and that run is the operator's to start.
- **Pointer**: the samples location rests on the case layout and run isolation in the
[eval suite reference](https://code.claude.com/docs/en/plugin-evals#eval-suite-reference) and on
the case loader in Claude Code 2.1.287, which reads only `prompt.md`, `case.yaml` and `graders/`.
The sample check grades as the binary does, per its grader code read against the
[grader types](https://code.claude.com/docs/en/plugin-evals#grader-types) table; Python's `re`
stands in for the JavaScript regex engine, and the script's header lists where the two differ.
- **As of**: 2026-10-01
- **Recheck trigger**: the next Claude Code release, a release that adds a file the loader reads
from a case directory, or a grader whose sample verdict disagrees with its verdict in a real run.
## When the parser stops
The script reads a bounded YAML subset: scalars, quoted scalars, flow lists and mappings, block
lists including lists of mappings, and block mappings three levels deep. An anchor, an alias, a tag,
a block scalar, a merge key, tab indentation, a nested sequence in either spelling, a backslash
escape outside `\\`, `\"`, `\/`, `\n`, `\t`, `\r`, or nesting past three levels is reported as
`frontmatter not parsed (<construct>)` at FAIL, never waved through at exit 0, because a validator
that green-lights input it could not read is worse than no validator. Simplify the file, or run the
CLI, which carries the full parser.
## Mirroring record
The load-time FAIL findings match what the binary rejects when it loads a case, so a suite at exit
0 loads.
- **Pointer**: the case schema recovered from the shipped Claude Code binary, read against the
eval-suite reference at <https://code.claude.com/docs/en/plugin-evals#eval-suite-reference>; the
required fields were re-derived from Claude Code 2.1.296 and each rejection reproduced with
`claude plugin eval --case`.
- **As of**: 2026-10-09
- **Recheck trigger**: the next Claude Code release, which can add a frontmatter key, add a grader
type, or move a bound. On a firing, re-derive both lists from the schema rather than patching
one value.
This skill asserts nothing about whether `claude plugin eval` is available in your install; it
reads files and never invokes anything. `/evals:plugin-eval` preflight owns that question.
## Boundary
This validator reads only the `prompt.md` / `case.yaml` / `graders/*.md` layout. The skill-creator
`evals/evals.json` format belongs to `/skill-quality:check validate-evals <skill>` when the
`skill-quality` plugin is installed; where it is not, that file goes unvalidated, since pointing
this validator at an `evals.json` reports only that the directory holds no cases.
## Next
`/evals:plugin-eval <target>`. A suite at exit 0 is one worth paying to run.
## Gotchas
- Exit 0 means the suite **loads**, not that it measures anything. A case whose graders pass in both
arms proves the plugin contributed nothing; the delta is read after a run, not here.
- Samples that all sort correctly prove the grader handles those answers, not the ones the model
will write. When a real run fails a correct answer, add that answer's phrasing as a sample.
- A case is a directory holding `prompt.md` or `case.yaml`. A directory holding neither is skipped
silently, so a misnamed prompt file reads as absent rather than as an error. An eval dir with no
case at all is a FAIL.
- `results/` and `mocks/` are never treated as cases, whatever they contain.
- When both files exist, `prompt.md` frontmatter overrides the matching `case.yaml` field, and the
bounds check reports the file the effective value came from, which may not be the file you edited.
- Unknown top-level keys in `case.yaml` are not reported. The unknown-key rejection is recorded for
`prompt.md`, and this validator invents no rejection the binary does not make.
- The script is Python 3.8+, standard library only, so it runs from a consumer checkout where no
package is provisioned. It never writes to the suite it reads.
Pliki w paczce (5)
- SKILL.md 11 KB
- evals/evals.json 5 KB
- scripts/test_validate_cases.py 47 KB
- scripts/validate-cases.py 60 KB
- scripts/validate-cases.test.sh 2 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · melodic-software/claude-code-plugins | 84 | MIT |
| DiogoHSM/driver | 100 | MIT |
| boshu2/agentops | 100 | Apache-2.0 |
| RemiPelloux/kill-my-idea | 100 | MIT |
| lalitmaganti/syntaqlite | 100 | Apache-2.0 |
| zanderadam/engy | 100 | MIT |
| hirogakatageri/hirokata-claude-marketplace | 100 | MIT |
| fabricio21x/fbs | 100 | MIT |
| Kiran-980/1ml | 100 | MIT |
Migawka bazy z 13.11.2026 · wersja paczki 2a3b3cdae67e (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 90write-case
Pisze lub poprawia przypadki testowe dla wtyczek, skilli i hooków Claude Code w oficjalnym formacie claude plugin eval. - 100add-eval
Dodaje lub edytuje przypadek ewaluacyjny wtyczki w plugin/evals (claude plugin eval): prompt, graderzy, mocki MCP, a potem uruchamia, porównuje i przebazowuje zestaw. - 86write-eval-case
Pisze jeden przypadek ewaluacji w formacie wtyczki Claude Code: prompt, skrypt przygotowania danych i deterministyczne oceny. - 100harness-eval
Pisze i uruchamia ewaluacje, które dowodzą, że harness działa lepiej niż bez niego, w formacie claude plugin eval z graderami i kosztami. - 100new-eval
Buduje zestaw ewaluacyjny skival do oceny jednego skilla taskmd: fixture, deterministyczne graders, cztery warianty, uruchomienie próbne i raport zapisany w repozytorium. - 95Evals
Liczy obiektywne metryki ewaluacji przez graderów kodowych, modelowych i ludzkich, z wynikami pass@k i pass^k, porównuje modele i prompty.