Przejdź do treści
Biblioteka skilli
AI i agenci · Testowanie

harness-eval

/harness-eval

Pisze i uruchamia ewaluacje, które dowodzą, że harness działa lepiej niż bez niego, w formacie claude plugin eval z graderami i kosztami.

Licencja
MIT
Ostatnia zmiana repozytorium
11.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
2
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 11.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Pisze i uruchamia ewaluacje, które dowodzą, że harness działa lepiej niż bez niego, w formacie claude plugin eval z graderami i kosztami.

Oryginalny opis (EN)

Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or harness, when running claude plugin eval, comparing with a baseline, or reading eval results.

Jak działa

  1. 1. Choose cases (gate)
  2. 2. Write cases
  3. 3. Check the suite before spending on it
  4. 4. Run (gate: cost)
  5. 5. Read results
  6. Testing always-loaded text (CLAUDE.md, rules)
  7. CI (document for the user; do not wire it yourself)
Sekcje z nagłówków SKILL.md, w oryginale · 109 linii, 877 słów
Podgląd SKILL.md
---
name: harness-eval
description: Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or harness, when running claude plugin eval, comparing with a baseline, or reading eval results.
---

# Harness evals

An eval case is a realistic prompt plus graders. `claude plugin eval` runs each case with the plugin
(WITH) and without it (W/OUT); Δ = WITH − W/OUT is what the harness adds. Full grader reference:
[references/grader-cookbook.md](references/grader-cookbook.md).

## 1. Choose cases (gate)

Start with 8–20 cases; quality beats count. Sources, best first:

1. Real failures: the research baseline probe, user reports, transcripts where plain Claude failed.
2. The work the harness exists for, phrased the way a user types it (never name the skill).
3. Should-not-fire cases: nearby requests the harness must leave alone (at least 1 in 5 cases).
4. One negative control: a task plain Claude already does well. It should show Δ ≈ 0, not harm.

Each case must be unambiguous: two experts would agree on pass/fail, and everything a grader checks
is stated in the prompt. Write the list (name, prompt, what passes, why plain Claude fails) and ask
the user to approve it.

## 2. Write cases

```
evals/<case-name>/
  prompt.md        frontmatter (run limits, tools) + the prompt
  graders/*.md     one grader per file
  case.yaml        only for fixtures: context.add_dirs, scaffold_script, history_file
```

```markdown
---
max_turns: 15
allowed_tools: [Read, Write, Edit, Glob, Grep, Skill]
tags: [core]
---
<the request, as the user would type it; name the output file path if a grader reads it>
```

Graders, per case: one on the **outcome** (a file or the final reply) and one on the **process**
(`tool_used` with `tool: Skill` — scored as a "plugin fired" indicator only). Prefer free graders:

| Check | Grader |
|---|---|
| a file was created | `file_exists` with `path:` glob |
| a produced file contains / lacks something | `regex` with `target: {source: file, path: <path>}`, `match: not_contains` for absence |
| the reply says something | `regex` on `last_message` |
| a tool was or was not used | `tool_used` (`min: 0`, `max: 0`, `arm: both` for must-not) |
| order of steps | `tool_order` |
| quality that needs judgment | `llm` with a short PASS/FAIL rubric on a short target |

Rules that keep scores stable: regex for long outputs; `llm` only on short text with concrete
PASS and FAIL lines; several small graders give partial credit; never check the exact path the
agent took when the outcome can be checked.

Platform limits: each run starts in an empty folder with no user config. On native Windows a run
cannot use Bash or PowerShell (WSL2 can, with `--allow-tools "Bash(<pattern>)"`). Fixtures go in
the case folder via `context.add_dirs` (read-only), or inline in the prompt.

## 3. Check the suite before spending on it

- `claude plugin eval <plugin> --case <name> --runs 1 --ablation none --max-cost-usd 1 --allow-tools Write Edit`
  on one or two cases: graders load and behave. A grader that fails on a correct result is a grader bug.
- A case at 0% in both arms over several runs is usually a broken case, not a hard one.
- Non-plugin targets: `hb wrap <target> .harness/wrapped --evals .harness/evals`, then evaluate
  `.harness/wrapped`. Re-run wrap after every change to the target.

## 4. Run (gate: cost)

1. `node "${CLAUDE_PLUGIN_ROOT}/scripts/hb.cjs" estimate <eval dir> --runs <n>` — put the range and
   the ceiling in your message to the user.
2. Run with the suggested ceiling. The plugin's hook stops the command for the user's approval and
   denies any run without `--max-cost-usd`. If the user declines, do not retry; ask what to change.
   `claude plugin eval <plugin> --runs <n> --max-cost-usd <ceiling> --allow-tools Write Edit --no-publish --trust-plugin`
   (`--trust-plugin` only for a plugin you or the user wrote; add `--keep-temp` to keep transcripts.)
3. `hb record <plugin>/evals/results --label "<what this run measures>"`.

Runs: 1 to iterate on graders; 3 before keeping or reverting any change. Pin `--model` when results
will be compared across days.

## 5. Read results

- Per case, not just the mean: a mean Δ can hide one regressed case.
- Δ > 0 with the Skill indicator passing: the harness helped. Δ ≈ 0 with WITH = 1.0: plain Claude
  already does this; the skill may be unnecessary. Δ < 0: suspect the judge or grader first
  (re-run with `--judge-model sonnet`), then the harness.
- Skill indicator failing: the description does not trigger on natural phrasing; fix the description.
- Read transcripts of failing runs (`--keep-temp`) before changing the harness. What the agent did
  not do is often the finding.
- Skip partial runs (`partial: true`, cost ceiling) in comparisons; `hb compare` warns about them.

## Testing always-loaded text (CLAUDE.md, rules)

The eval arms differ only by plugin, and CLAUDE.md does not load from a plugin. To measure CLAUDE.md
text, compare two single-arm runs:

```
hb variant <plugin> --from evals --to evals-ctx --append .harness/wrapped/context.md
claude plugin eval <plugin> --ablation none --runs 3 ...                     # without the text
claude plugin eval <plugin> --eval-dir evals-ctx --ablation none --runs 3 ... # with the text
```

Record both and `hb compare` them. `append_system_prompt` sits in the system prompt, where real
CLAUDE.md text does not, so report this as an approximation.

## CI (document for the user; do not wire it yourself)

`claude plugin eval . --trust-plugin --json results.json --threshold 0.8 --model <m> --judge-model <m> --no-publish --max-cost-usd 20`
Exit 0 pass, 1 below threshold or error, 2 partial (cost ceiling).
Pliki w paczce (2)
  • SKILL.md 6 KB
  • references/grader-cookbook.md 4 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · mhmdsalameh/ai-native-harness100MIT
tech-leads-club/agent-skills100MIT

Migawka bazy z 13.11.2026 · wersja paczki cfa16645cd05 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100add-eval
    Dodaje lub edytuje przypadek ewaluacyjny wtyczki w plugin/evals (claude plugin eval): prompt, graderzy, mocki MCP, a potem uruchamia, porównuje i przebazowuje zestaw.
  • 90write-case
    Pisze lub poprawia przypadki testowe dla wtyczek, skilli i hooków Claude Code w oficjalnym formacie claude plugin eval.
  • 86write-eval-case
    Pisze jeden przypadek ewaluacji w formacie wtyczki Claude Code: prompt, skrypt przygotowania danych i deterministyczne oceny.
  • 84plugin-eval
    Prowadzi przez praktykę z claude plugin eval: kontrola wstępna, walidacja bez kosztów, szacunek kosztu, uruchomienie i odczyt różnicy między wynikiem ze skillem i bez niego.
  • 100skill-eval
    Mierzy, czy skill faktycznie pomaga: buduje zestaw testów plugin eval, uruchamia go z limitem wydatków i porównuje wyniki z wersją bez skilla.
  • 84validate
    Waliduje statycznie zestaw claude plugin eval (prompt.md, case.yaml, graders/*.md) przed uruchomieniem kosztownych przebiegów: błędy ładowania, ostrzeżenia o typowych pomyłkach i offline sprawdzenie graderów.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.