new-eval
/new-eval
Buduje zestaw ewaluacyjny skival do oceny jednego skilla taskmd: fixture, deterministyczne graders, cztery warianty, uruchomienie próbne i raport zapisany w repozytorium.
- Autor
- driangle ↗
- Repozytorium
- taskmd ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 02.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 02.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Buduje zestaw ewaluacyjny skival do oceny jednego skilla taskmd: fixture, deterministyczne graders, cztery warianty, uruchomienie próbne i raport zapisany w repozytorium.
Oryginalny opis (EN)
Build a skival eval suite that benchmarks one taskmd skill — fixture, deterministic graders, four variants, smoke run, and a committed report. Use when adding a suite under evals/, benchmarking a skill, or measuring whether a skill actually helps.
Jak działa
- Phase 0 — Decide what you are measuring
- Phase 1 — Fixture
- Phase 2 — Choose verifiers
- Phase 3 — Write the grader module
- Phase 4 — suite.yaml
- Phase 5 — Smoke run
- Phase 6 — Full run
- Phase 7 — Audit, then write it up
- Reporting honestly
Podgląd SKILL.md
--- name: new-eval description: Build a skival eval suite that benchmarks one taskmd skill — fixture, deterministic graders, four variants, smoke run, and a committed report. Use when adding a suite under evals/, benchmarking a skill, or measuring whether a skill actually helps. metadata: internal: true --- # New eval suite Build a suite under `evals/<skill>/` that answers one question: **does this skill make an agent better than a vanilla session in the same project, and is it worth what it costs?** `$ARGUMENTS` is the skill to benchmark (e.g. `next-task`). Read [`evals/CLAUDE.md`](../../../evals/CLAUDE.md) and [`evals/README.md`](../../../evals/README.md) first. Two reference implementations, and you should copy from whichever matches your skill: | | Skill kind | Grades | Copy for | |---|---|---|---| | `evals/add-task/` | **writes** task files | the filesystem (`check`) | mutating skills | | `evals/list-tasks/` | **reads** and reports | the agent's output (`check_output`) | read-only skills | **Runs cost ~$20 and 20 minutes. Ask before starting a full one.** ## Phase 0 — Decide what you are measuring 1. Read the shipped skill in **both** plugins — `claude-code-plugin/skills/<skill>/SKILL.md` and `claude-code-plugin-lite/skills/<skill>/SKILL.md`. The suite compares them, so know how they differ (the full plugin drives the CLI; lite reads files directly). 2. Classify it: does it **write** or **report**? That decides the grading route and everything downstream. 3. Check `tasks/` for an existing task describing the benchmark — it may already specify the evals, the fixture, and the expected sets. 4. Sketch 4–6 evals before writing any YAML. **One behavior per eval** — an eval asserting two things lets one failure mask the other. Each needs a crisp, deterministic answer; if you cannot state the expected result as a set of task IDs, the eval is not gradable. ## Phase 1 — Fixture Fork the shared base — copy it, do not point `dir:` at it, so suites stay independently editable: ```bash cp -R evals/fixtures/workspace evals/<skill>/workspace cp -R evals/fixtures/workspace-bare evals/<skill>/workspace-bare ``` - Bring your **own** fixture only if the skill needs adversarial state (`validate-tasks` needs seeded defects, `verify-task` needs `verify:` blocks, `complete-task` needs `workflow: pr-review`). Say so in the suite's task file. - **Never add tasks to `evals/add-task/workspace/`** — it is frozen at five. - **Verify ground truth from a copy outside this repo.** Inside it, `taskmd list` returns taskmd's own 200+ tasks while `taskmd validate` correctly reports six. Record the exact queries and expected IDs; `evals/fixtures/README.md` has the current set. ## Phase 2 — Choose verifiers Every eval gets `agent_exits_ok` plus `tool_not_used` (hermeticity), then: - **Write skill** → `check`, a Go grader inspecting the created task file. - **Read-only skill** → `check_output`, which pipes the agent's final text to a grader on **stdin**, plus a `no-mutation` `check` asserting the fixture is untouched. That is the real failure mode of a read skill: an agent that "helpfully" edits while answering. Three rules that decide whether the grader measures anything: - **Assert two-sided.** `output_contains` is presence-only, so an agent that ignores the filter and dumps everything passes a "pending only" eval. - **Match IDs and field names, never phrasing or layout.** Grading the plugin skill's pretty-printed table measures formatting compliance and unfairly fails `no-skill`. - **Let structure separate results from commentary.** Agents end correct filtered answers by naming what they excluded. Count only list-item lines when the answer has two or more; treat prose as commentary. Do not blocklist cue words — that was tried and lost. ## Phase 3 — Write the grader module Start from `evals/list-tasks/workspace/.verify/` — stdlib only, its own `go.mod` (kept out of `go.work`, hence `GOWORK=off`), a `checks` map of name → func, one func per eval. Then, before spending anything: ```bash cd evals/<skill>/workspace/.verify GOWORK=off go test ./... # both directions, zero tokens cp -R . ../../workspace-bare/.verify/ # the two copies must stay identical ``` Write `output_test.go` covering realistic agent answers in **both** directions — a table, a bullet list, a chatty answer that explains its exclusions, and a dump-everything answer that must *fail*. **A check that cannot fail measures nothing.** For filesystem checks, hand-test from a temp copy outside the repo: build the expected state, confirm PASS, break one thing, confirm FAIL. ## Phase 4 — suite.yaml Copy the structure from `evals/list-tasks/suite.yaml`: - `defaults`: runner, model, `samples`, `timeout`, `parallel`, and `runner_config` pinning **`allowed_tools`** — skival compiles it into the CLI's `--tools` flag, an exclusive whitelist over built-ins, so every tool you do not list (`Write`, `Skill`, `TaskCreate`) is denied at registration. Get this list right and the suite is hermetic; get it wrong and every variant silently invokes your own installed plugin skill and the suite measures nothing. `disallowed_tools` is advisory only — keep it as documentation of intent, but it is not what enforces. **If your suite has a `no-mutation` check, `Write` and `Edit` must be in `allowed_tools`**, or the check cannot fail and proves nothing. - `ranking` weights, and the four variants via a `&variants` anchor: `no-skill`, `plugin-skill`, `lite-skill`, `bare-project` (which overrides `dir:` and shadows `taskmd` on `PATH`). - `isolate: true` on every eval. - Verify order: `agent_exits_ok` → `tool_not_used` → `no-mutation` → correctness. The pipeline short-circuits, so put the safety checks first. - Write `run:` as `/bin/sh -c '…'`. Then `skival validate suite.yaml` — free, and catches structural mistakes. ## Phase 5 — Smoke run ```bash ./evals/run-eval.sh <skill> --samples 1 ``` ~5% of the cost, and **this is where grader bugs surface** — the `list-tasks` smoke run caught two (one harness, one grader), and a third turned up later while classifying failures. Read the actual agent outputs, not just the pass/fail column. Re-grade failures by hand and ask whether the agent was wrong or the grader was. Fix and re-smoke until every verdict is one you can defend. ## Phase 6 — Full run 1. **Commit first** — a dirty tree is recorded against a commit it did not measure. 2. Report the smoke result and the estimated cost, and **ask before spending**. 3. `./evals/run-eval.sh <skill>` 4. **Never report a run that did not finish.** Errored samples are recorded `"pass": null` and dropped from the math while the rankings still print. ## Phase 7 — Audit, then write it up - Read every failing output and confirm it is a genuine agent failure. Graders have bugs too. - Cross-check the tool census against `allowed_tools` for leakage. - `evals/<skill>/reports/<date>-<commit>.md` — the durable per-run record. Never overwrite a past run's report. - `evals/<skill>/REPORT.md` — cross-run index, one row per run with its commit. - `evals/<skill>/SUGGESTIONS.md` — grounded in observed failures only. Each item names the failure it fixes and how many samples it would flip. No speculative polish. - Add the suite to the table in `evals/README.md`, and a per-suite `CLAUDE.md` if it has traps a future agent would hit. ## Reporting honestly - **Do not attribute a delta to your change without a mechanism.** Variants with unchanged inputs have drifted 17.5 points between runs. A pass-rate swing plus a named cause plus a targeted eval moving is evidence; a swing alone is not. - If you edited a skill and its score moved, **A/B it against the previous version** before calling it a regression or a win. - State grading judgment calls explicitly — where an agent's answer was defensible but graded wrong, say so and say how many samples it affects.
Wersje
W bazie nie ma innych skilli o nazwie new-eval.
Migawka bazy z 13.11.2026 · wersja paczki e2fbc5368eb9 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100add-eval
Dodaje lub edytuje przypadek ewaluacyjny wtyczki w plugin/evals (claude plugin eval): prompt, graderzy, mocki MCP, a potem uruchamia, porównuje i przebazowuje zestaw. - 100skill-eval
Mierzy, czy skill faktycznie pomaga: buduje zestaw testów plugin eval, uruchamia go z limitem wydatków i porównuje wyniki z wersją bez skilla. - 84validate
Waliduje statycznie zestaw claude plugin eval (prompt.md, case.yaml, graders/*.md) przed uruchomieniem kosztownych przebiegów: błędy ładowania, ostrzeżenia o typowych pomyłkach i offline sprawdzenie graderów. - 86write-eval-case
Pisze jeden przypadek ewaluacji w formacie wtyczki Claude Code: prompt, skrypt przygotowania danych i deterministyczne oceny. - 100create-skill-test
Tworzy specyfikacje ewaluacji eval.yaml dla skilli, własnych agentów i paczek workflow gh-aw w repozytorium dotnet/skills, w tym bodźce, graderów, rubryki i pliki fixture. - 100harness-eval
Pisze i uruchamia ewaluacje, które dowodzą, że harness działa lepiej niż bez niego, w formacie claude plugin eval z graderami i kosztami.