Przejdź do treści
Biblioteka skilli
AI i agenci · Testowanie

harness-evaluation

/harness-evaluation

Buduje i uruchamia oceny bazowe i kandydackie dla instrukcji agentów i skilli, diagnozuje błędy oceny i mierzy poprawę.

Licencja
MIT
Ostatnia zmiana repozytorium
13.11.2026

Ocena i bezpieczeństwo

90/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść25 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
10
Skrypty
5
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 13.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 5 skryptów w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/mine_traces.py, scripts/run_evaluation.py, scripts/test_mine_traces.py, scripts/test_run_evaluation.py, scripts/test_run_evaluation_memory.py

Co robi i kiedy

Buduje i uruchamia oceny bazowe i kandydackie dla instrukcji agentów i skilli, diagnozuje błędy oceny i mierzy poprawę.

Oryginalny opis (EN)

Build and run baseline/candidate evaluations for agent instructions and skills, diagnose evaluation failures, and measure supported improvements. Use for harness maintenance, not ordinary application testing.

Jak działa

  1. Choose evidence that answers the claim
  2. Diagnose traces and choose informative tasks
  3. Discover the evaluation target
  4. Construct and run
  5. Recover from evaluation failures
  6. Decide and report
Sekcje z nagłówków SKILL.md, w oryginale · 127 linii, 1 114 słów
Podgląd SKILL.md
---
name: harness-evaluation
description: Build and run baseline/candidate evaluations for agent instructions and skills, diagnose evaluation failures, and measure supported improvements. Use for harness maintenance, not ordinary application testing.
---

# Harness evaluation

Requires `ahl` 0.169.0 or later; `ahl doctor` warns when the CLI or these skill links are stale (fix: `ahl update`).

Evaluation is an executable part of harness improvement. Construct missing tests,
run them, inspect outcomes, and repair the evaluation when necessary. Do not require
an existing benchmark, perfect labels, complete history, or statistical significance
before doing a useful bounded comparison. Never turn limited evidence into a broader
claim than it supports.

## Choose evidence that answers the claim

| Evidence | Establishes | Does not establish |
| --- | --- | --- |
| Schema, references, source/contract checks | Structural validity and verified contradiction removal | Model behavior or task success |
| Read-only decision probe | Decisions on the supplied scenario | Execution, deployment, or actual side effects |
| Task execution and artifact grader | Observed completion under the recorded conditions | General reliability or production acceptance |
| Blinded rubric over actual outputs | Qualified comparative quality | Objective correctness or independent replication |

Prefer external executable checks of final state. Model judgment supplements them;
require concrete artifact/trace citations and hide version labels from the judge.
Do not grade success from the agent's own final claim or source-word matching.

## Diagnose traces and choose informative tasks

For multi-session diagnosis or recurring failures, read
[trace diagnosis and method selection](references/trace-diagnosis.md). Use process
mining to discover sequences and case-supported patterns, then check raw evidence
and current harness contracts. Patterns propose mechanisms; they do not establish
causation or authorize a change. Use ML only when it answers a named question
better than simple counts and a reviewed sample.

Cover distinct material mechanisms, not just several variants of the first one.
Freeze an observable acceptance boundary for each selected mechanism. Include a
partial repair that the grader must reject, a successful control and a relevant
safety regression. Preserve a held-out task family or time period when testing
transfer; random event splits can leak the same session into both sets.

If baseline and candidate both pass, distinguish a redundant rule from an
insensitive test: did the baseline exercise the suspected failure, do ambient
instructions already supply the fix, and would a blocker-only or superficial
repair pass the grader? Strengthen one test from the actual failure mechanism
without encoding the solution. A tie closes that candidate, not unrelated
mechanisms. Do not spend the entire budget repeatedly confirming easy passes.

## Discover the evaluation target

A user-provided failure or candidate is optional. Inspect analysis results,
recurring patterns, failed evaluations and current contracts to identify what to
test. Use `ahl analyze` for scoped evidence collection when available. Unknown
capture remains a limitation, not success or proof of failure. In a CLI-managed
run the parent owns model execution and adoption; prepare its requested frozen
manifest and grader controls without launching recursive controllers.

## Construct and run

1. Inspect available runtime, authentication status, existing tests and fixtures.
   Reuse installed tools. The bundled [runner](scripts/run_evaluation.py) supports
   Codex and Claude Code JSONL trials; read [the execution contract](references/execution.md) to use it.
2. Freeze task inputs and graders before scoring. Start with the observed failure,
   an adjacent regression, and an unseen transfer case; separate development and
   held-out feedback. Synthetic cases must represent the failure mechanism, not
   contain its solution. A historical approximation is labeled as such.
3. Snapshot baseline/candidate files and record hashes, model, reasoning settings,
   tool access, ambient instructions and resource limits. Keep conditions matched
   within each pair; record unavailable controls. Never pool different models as
   comparable repeats. Do not change model or permissions to chase a better score.
4. Use fresh isolated fixture copies for each trial. Keep candidate instructions out
   of baseline, graders and reference answers outside writable task workspaces, and
   evaluation artifacts outside skill discovery. Verify the intended version is
   loaded; [trial isolation](references/isolation.md) lists what a trial still
   inherits. Worktrees, if needed, belong under `~/dev/.worktrees/`.
5. Run baseline, then candidate on identical inputs; preserve negative results.
   Inspect artifacts and traces, not just exit codes. Repeat the decisive behavioral
   difference in fresh sessions. Reserve unseen tasks until candidate selection;
   if their feedback informs another edit, use new held-out cases.
6. Assess success and critical regressions first. Record time, tool calls and token
   usage only when observed. Compare efficiency with matched task and feedback
   budgets; extra retries do not prove a better harness. If both pass, do not claim
   a success-rate gain. Test a harder relevant case or retain the simpler version.

Default shared budget: 24 agent trials and 45 minutes including recovery and nested
trials. Within it, allow at most three candidate variants and one refinement per
mechanism; do not stop a multi-mechanism audit after three total fixes. Allocate
coverage before repetitions and reserve about a quarter for final checks. These
ceilings are not targets. Use user-specified limits when supplied.

## Recover from evaluation failures

Use this order within the authorized scope; do not ask the user to design the eval:

1. Diagnose the failed command or missing artifact. Distinguish task failure from
   runner/setup failure, grader defects, timeout, unavailable access and invalid data.
2. Repair local setup and retry a transient failure once. An unrelated connector
   startup warning does not invalidate an otherwise completed trial.
3. Use another already-authorized runtime when available; rerun both sides under
   that runtime. Never inherit candidate conclusions into a supposedly fresh baseline.
4. Construct a smaller executable reproduction with controlled fixtures and an
   observable oracle. Missing production access narrows claims to local behavior.
5. For decision-only claims, use matched decision probes. For qualitative outputs,
   compare actual executed outputs with a frozen blinded rubric and report uncertainty.

No usable runtime at all is a real execution error, not evidence of success. Continue
independently checkable corrections and report the exact failed capability and recovery
attempts. Missing history, a large benchmark, perfect controls, or labels alone must
never terminate the improvement workflow.

Version a repaired grader and rerun both sides. Do not let the candidate edit graders,
reference answers or thresholds. Missing completion events, malformed traces, timeouts,
and missing results cannot pass. A passing process with a failed acceptance check is
still a task failure.

## Decide and report

Recommend adoption only for the claim supported by the checks: targeted behavioral
benefit with relevant regressions passing, or a directly verified contract correction.
Reject critical correctness or safety regressions. Retain the simpler baseline when
no benefit is demonstrated. Structural checks alone do not support behavioral claims.

Report versions, task roles, actual outcomes, failed/not-executed checks, sample size,
conditions, observed costs and limitations. Report tested, rejected, already-covered
and deferred mechanisms separately; data volume is not a coverage measure.
Keep sanitized traces and rollback artifacts;
do not create routine ai-memory pages. Research informs experiment design, not local
proof: see [evidence and limits](references/evidence-to-design.md).
Pliki w paczce (10)
  • SKILL.md 8 KB
  • references/evidence-to-design.md 7 KB
  • references/execution.md 28 KB
  • references/isolation.md 3 KB
  • references/trace-diagnosis.md 11 KB
  • scripts/mine_traces.py 2 KB
  • scripts/run_evaluation.py 67 KB
  • scripts/test_mine_traces.py 3 KB
  • scripts/test_run_evaluation.py 38 KB
  • scripts/test_run_evaluation_memory.py 13 KB

Wersje

W bazie nie ma innych skilli o nazwie harness-evaluation.

Migawka bazy z 13.11.2026 · wersja paczki b47d920115db (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100eval-harness
    Mierzy wydajność agentów, wskazuje słabe strony i śledzi poprawę w czasie, przy ocenie jakości agentów, porównywaniu podejść i walidacji zmian.
  • 100eval-harness
    Mierzy wydajność agentów, wskazuje słabe strony i śledzi poprawę w czasie, przy ocenie jakości agentów, porównywaniu podejść i walidacji zmian.
  • 100evaluation
    Buduje frameworki ewaluacji systemów agentowych, aby testować wydajność agentów, walidować decyzje context engineering i mierzyć poprawę w czasie.
  • 100evaluation
    Buduje frameworki ewaluacji systemów agentowych, aby testować wydajność agentów, walidować decyzje context engineering i mierzyć poprawę w czasie.
  • 100evaluation
    Buduje frameworki ewaluacji systemów agentowych, aby testować wydajność agentów, walidować decyzje context engineering i mierzyć poprawę w czasie.
  • 100edit-skills
    Ustala konwencje dla instrukcji agentów i skilli w repozytorium, przy dodawaniu lub audycie skilla.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.