skill-testing
/skill-testing
Sprawdza, czy skill jest poprawny, uruchamia się przy właściwych zapytaniach i działa lepiej niż brak skilla.
- Autor
- patrykmurzyn ↗
- Repozytorium
- skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 28.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 10
- Skrypty
- 3
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 28.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
- 3 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.
Skrypty: scripts/micro_test.py, scripts/trigger_eval.py, scripts/validate_skill.py
Co robi i kiedy
Sprawdza, czy skill jest poprawny, uruchamia się przy właściwych zapytaniach i działa lepiej niż brak skilla.
Oryginalny opis (EN)
Tests whether an agent skill is valid, loads on the right requests and does better than no skill or its previous version. Use when checking, evaluating or debugging a skill. Step 3 of `skill-authoring`.
Jak działa
- 1. Validate (free, every change)
- 2. Micro-test the wording (behaviour-shaping lines)
- 3. Trigger evals
- 4. Output evals
- 5. Pressure tests (discipline skills only)
- 6. Iterate
- Gotchas
Podgląd SKILL.md
---
name: skill-testing
description: Tests whether an agent skill is valid, loads on the right requests and does better than no skill or its previous version. Use when checking, evaluating or debugging a skill. Step 3 of `skill-authoring`.
license: MIT
metadata:
author: patrykmurzyn
version: "0.2.1"
status: experimental
---
# Skill testing
A skill that reads well can still be invalid, never load, load on the wrong requests, or do no better than no skill. Test those layers in order and stop at the first that fails. Every run that starts an agent session costs money: tell the user how many sessions a step needs and get a yes first.
## 1. Validate (free, every change)
```sh
uv run scripts/validate_skill.py <skill-dir>...
```
It checks the spec, size limits (500 lines, 20,000 bytes), unfinished placeholders, broken links, hidden Unicode, blank-line padding, HTML comments and secret-like strings, and reports duplicate names. Add `--strict-spec` for skills that may go to claude.ai. Inside a skills repo, run the repo's `scripts/validate.sh` instead. Fix every error and read every warning.
## 2. Micro-test the wording (behaviour-shaping lines)
Before a full eval, test a new or disputed instruction on its own: the same prompt with and without the line, five or more fresh runs per arm, every output read by hand. If the control arm doesn't show the failure, the line isn't needed. If results vary a lot between runs, the wording is ambiguous. Read [micro-tests](references/micro-tests.md) for the script and the procedure.
## 3. Trigger evals
Write `evals/trigger-queries.json` in the skill under test:
```json
{"skill": "<name>", "queries": [
{"query": "...", "should_trigger": true, "split": "train"}
]}
```
- About 20 realistic queries: half should load the skill, half are near misses that share its words but need something else. Use file paths, casual tone, typos and the user's own language. Add the pre-mortem cases from the brief.
- Put about 40% in `"split": "validation"` and never tune the description on them.
- Should-trigger queries must need the skill. Agents skip skills for simple one-step requests, however well the description matches.
```sh
uv run scripts/trigger_eval.py <skill>/evals/trigger-queries.json --agent claude --runs 3
uv run scripts/trigger_eval.py <skill>/evals/trigger-queries.json --agent codex --runs 3
```
Use `--limit 2 --runs 1` for a cheap smoke test and `--model` to test each model the skill will run on, smaller models included. A query passes at a trigger rate of 0.5 or more if it should load the skill, and below 0.5 if it shouldn't. Fix failures through `skill-writing`: widen when it applies if it under-triggers, add the boundary if it false-triggers. Keep the version with the best validation pass rate, which is not always the last; stop after about five rounds.
## 4. Output evals
Run each case in a fresh context with the skill, and again without it, or with the snapshot of the previous version when reworking. Give the runner the request and the raw inputs only, never the expected answer or the suspected bug. Grade results and process, review the assertions themselves, and pin the model so a model change isn't mistaken for a skill change. Read [the eval guide](references/evals.md) before the first run.
## 5. Pressure tests (discipline skills only)
For a skill whose kind is *discipline*, a calm test proves little: the agent knows the rule and breaks it when rushed. Run pressure scenarios and record the excuses word for word. Read [pressure tests](references/pressure.md) for the method.
## 6. Iterate
Hand `skill-writing` the failed assertions, the user's feedback and what the transcripts show:
- logic every run rewrote, which becomes a bundled script;
- instructions the agent followed though they didn't apply, which get cut;
- reference files it never opened or kept re-reading, which get moved or merged.
Generalize instead of patching single cases, keep the skill lean, bump `metadata.version`, and re-run. Stop when feedback is empty or the gains stop. Set `status: stable` only after the evals pass.
## Gotchas
- Your own session is not a fresh context. You know what the skill should do, so your run proves nothing; test in a new one.
- `trigger_eval.py` runs each query in an empty temp directory, so project `AGENTS.md` files and project skills don't leak into the result. The skill under test must be installed globally.
- Codex trigger detection is heuristic: it looks for the skill's SKILL.md in Codex's item events. Confirm surprising results in a transcript.
- A grader that only checks "the skill was used" inflates the with-skill score; leave it out of the with/without comparison.
- A skill with identical results with and without it adds nothing. Cut it or rescope it with `skill-scoping`.
Pliki w paczce (10)
- SKILL.md 5 KB
- evals/evals.json 1 KB
- evals/files/README.md 1 KB
- evals/files/broken-skill/SKILL.fixture.md 1 KB
- references/evals.md 4 KB
- references/micro-tests.md 2 KB
- references/pressure.md 2 KB
- scripts/micro_test.py 5 KB
- scripts/trigger_eval.py 8 KB
- scripts/validate_skill.py 15 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · patrykmurzyn/skills | 100 | MIT |
| javiarmesto/aldc-al-development-collection | 100 | MIT |
| javiarmesto/aldc-al-development-collection | 100 | MIT |
| javiarmesto/aldc-al-development-collection | 100 | MIT |
| javiarmesto/aldc-al-development-collection | 100 | MIT |
| latestaiagents/agent-skills | 100 | MIT |
Migawka bazy z 13.11.2026 · wersja paczki 77b98bdf7da0 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100skill-authoring-lifecycle
Tworzy szkielet i formalnie przegląda nowy Agent Skill w repozytorium marketplace: poprawny SKILL.md, układ katalogów, wpisy w rejestrze i audyt specyfikacji przed PR. - 90write-a-skill
Tworzy, sprawdza i poprawia skille agentów oraz decyduje, czy wskazówkę lepiej zapisać jako skill, polecenie, hook czy regułę. - 100skill-authoring
Pomaga przy tworzeniu nowego skilla albo workflow oraz poprawie skilla, który się nie uruchamia lub nie jest wykonywany rzetelnie. - 100skill-authoring-lifecycle
Tworzy szkielet nowego skilla w repozytorium marketplace i formalnie go sprawdza: frontmatter, układ katalogów i gotowość do PR. - 100skill-tune
Diagnozuje, dlaczego skill nie uruchamia się lub działa na złe zapytania, i przepisuje jego opis. - 100skill-eval
Uruchamia i analizuje przypadki testowe ewaluacji skilli Claude, sprawdza, czy skill działa i jaka jest jego jakość, także po stworzeniu lub zmianie skilla.