Skip to content
Skills Library
AI & agents · Building & coding

create-eval

/create-eval

How to build or extend an eval in MudPuppy the same way everywhere — Anthropic's eval terms (G-33), the evals/ folder layout (tasks, prompts, harness, graders by kind, results), naming, kinds of task (map, trace, live site), writing ground truth in YAML (G-34), live-site safety, and the rules for graders, trials, conditions and baselines. Use when creating an eval for any app or agent (e.g. the chat agent), adding a task, a ground truth, a grader or an LLM judge, running or promoting trials, or restructuring eval code. Not for deterministic tests or gates (those are tests, not evals).

Repository
mudpuppy_ai ↗
Licence
Apache-2.0
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

How to build or extend an eval in MudPuppy the same way everywhere — Anthropic's eval terms (G-33), the evals/ folder layout (tasks, prompts, harness, graders by kind, results), naming, kinds of task (map, trace, live site), writing ground truth in YAML (G-34), live-site safety, and the rules for graders, trials, conditions and baselines. Use when creating an eval for any app or agent (e.g. the chat agent), adding a task, a ground truth, a grader or an LLM judge, running or promoting trials, or restructuring eval code. Not for deterministic tests or gates (those are tests, not evals).

Original author description

How to build or extend an eval in MudPuppy the same way everywhere — Anthropic's eval terms (G-33), the evals/ folder layout (tasks, prompts, harness, graders by kind, results), naming, kinds of task (map, trace, live site), writing ground truth in YAML (G-34), live-site safety, and the rules for graders, trials, conditions and baselines. Use when creating an eval for any app or agent (e.g. the chat agent), adding a task, a ground truth, a grader or an LLM judge, running or promoting trials, or restructuring eval code. Not for deterministic tests or gates (those are tests, not evals).

How it works

  1. 1. Terms (G-33)
  2. 2. Folder layout
  3. 3. Naming
  4. 4. Kinds of task
  5. 5. Ground truth (G-34)
  6. 6. Live sites (a production system we don't own)
  7. 7. Graders
  8. 8. Trials, conditions and baselines
  9. 9. Wiring and checks
  10. 10. New eval checklist
Original SKILL.md headings · 199 lines, 1,889 words
SKILL.md preview
---
name: create-eval
description: How to build or extend an eval in MudPuppy the same way everywhere — Anthropic's eval terms (G-33), the evals/ folder layout (tasks, prompts, harness, graders by kind, results), naming, kinds of task (map, trace, live site), writing ground truth in YAML (G-34), live-site safety, and the rules for graders, trials, conditions and baselines. Use when creating an eval for any app or agent (e.g. the chat agent), adding a task, a ground truth, a grader or an LLM judge, running or promoting trials, or restructuring eval code. Not for deterministic tests or gates (those are tests, not evals).
---

# Create an eval

The reference implementation is `apps/crawler/evals/` (read its `README.md` first). A new eval
copies its shape; a difference needs a reason written in the new eval's README. The order of work
(what to decide on paper before any code) is `learning/10-writing-evals.md`; writing ground truth
is `learning/11-writing-ground-truth.md`.

## 1. Terms (G-33)

Use Anthropic's vocabulary ("Demystifying evals for AI agents") in code, docs and messages:

| Term | Meaning | Here |
|---|---|---|
| Task | One test: defined inputs and success criteria | An entry in `SUITE`, named by its portal id |
| Evaluation suite | The set of tasks | `SUITE` in `tasks.ts` |
| Trial | One attempt at a task; outputs vary, so run several | One `pnpm eval` run, one result file |
| Grader | Logic that scores one aspect: code-based, model-based (LLM judge) or human | `graders/<kind>/` |
| Transcript | The full record of a trial: outputs, tool calls, reasoning | The agents' CLI session files |
| Outcome | The final state of the environment after the trial | The workspace store and documentation |
| Evaluation harness | The code that runs evals end to end | The `evals/` folder |
| Agent harness | What lets the model act as an agent | MCP servers, Claude Code |
| Reference solution | A known working output that passes all graders | Hand-made, proves graders work |

Ours, kept on purpose: *ground truth* (facts the graders compare against), *baseline* (the trial
chosen as the standard) and *tolerance* (allowed drop). Domain terms of the system under test
(crawl run, state, frontier, persona) are not eval terms. A new concept takes Anthropic's name when
one exists; coin a term only when they have none.

## 2. Folder layout

At the root of the app or package the eval measures: `apps/<app>/evals/`, never under `scripts/`
or inside a sub-package.

```
evals/
  README.md        terminology, layout, graders table, kinds of task, where data lives, known gaps
  index.ts         entry: parse config → preflight → task → dry run or runTrial(); reads top to bottom
  config.ts        parseArgs (strict: unknown flags are errors) into one EvalConfig; USAGE
  failures.ts      die (before the trial), EvalAbort + abort (inside it, so `finally` cleans up)
  tasks.ts         EvalTask, TraceSpec, SUITE, taskFor(), loadGroundTruth(): the suite as data
  ground-truth/    <task>.yaml for systems we don't own (G-34)
  prompts.ts       PROMPTS: what each agent is asked (map, trace, BA); the *how* stays in the
                   agent definitions and skills
  queries.ts       named read-only store queries, one function per question
  results.ts       TrialRecord, buildResult(), saveResult(), promote, comparability warnings
  harness/         index.ts + one file per concern: workspace, <environment>-server, agent,
                   cost, preflight
  graders/
    index.ts       grade() and what index.ts needs
    grade.ts       grade(): runs every grader over one finished trial
    code/          code-based graders, one file per thing graded
    model/         model-based graders (LLM judge): the mechanism + judges/<id>.ts, one per judge
    human/         human labels and review flows: labels.ts + labels/<judge>.yaml
  tests/           unit tests of every pure module (fast tier)
```

Keep all three grader folders even when empty, so the kinds are visible.

## 3. Naming

- Files name their concern (G-05): `docs-coverage.ts`, `crawl-coverage.ts`, `process-trace.ts`,
  `leaks.ts`, `run-checks.ts`; never `utils.ts`/`helpers.ts`.
- Types and values: `EvalConfig`, `EvalTask`, `TraceSpec`, `SUITE`, `taskFor`, `TrialRecord`,
  `Grades`, `GradeInput`. Functions: `runTrial()`, `grade()`, goal graders `score<Thing>()`,
  guardrails `sweep<Thing>()` or `<thing>Checks()`, queries named by the question
  (`pendingFrontierCount`, `traceStepRoutes`).
- Result fields: `scores.<category> = { matched, total, pct }` (e.g. `crawl_screens`,
  `trace_steps`, docs categories), `overclaims`, `checks.<name>` (booleans or counts, e.g.
  `crawler_completed`, `trace_goal_reached`, `password_hits`), `conditions.<name>`.
- A task is named by its portal id; its results go to `data/evals/<task>/`, its ground truth to
  `<task>.yaml` (or `ground-truth.yaml` next to a portal we own).
- An anonymous persona used by an eval is named `guest`: crawl coverage treats any other name as
  logged in (known gap, see the reference README).
- No nested ternaries (lint `no-nested-ternary`): choices between tasks are data in `tasks.ts`.
- No module-level globals besides constants: pass `EvalConfig` (and the task, the trial's
  `startedAt`) down.
- Folders that hide internals export through `index.ts` (G-12).

## 4. Kinds of task

| Kind | In `tasks.ts` | The agent does | Extra grading |
|---|---|---|---|
| Map (default) | no `trace` | maps the whole portal | — |
| Map with process traces | `processTraces: [{ processId, name, goal }]` | maps, then traces each listed process (guest only; `--no-process-traces` skips) | `process_trace_steps`, `process_traces_goal_reached`; condition `process_traces` |
| Trace | `trace: { processId, name, goal }` | walks one named process toward the goal (trace mode) | `trace_steps` (the ground-truth process's `steps` reached in order), `trace_goal_reached` |
| Live site | no `server` | as above, on a site we don't own; nothing starts locally | — (see §6) |

A trace task's ground-truth process must list its `steps` (routes in order); a test checks this.

## 5. Ground truth (G-34)

- **YAML, never JSON**, starting with a comment naming the portal and the source of the facts. Read
  with `readGroundTruth()` (`@mudpuppy/docs`), validated by the `GroundTruth` schema.
- **Where**: next to the system when we own it (its self-check tests read it there); in
  `evals/ground-truth/<task>.yaml` when we don't. `tasks.ts` names the path either way.
- **Written by a person, not from agent output.** The person writes notes (fact, `how_i_know`,
  `who_sees`, `counts_when`, `unsure:`); the schema file is produced from them. Agent output may be
  a starting point only for a site we don't own, and only after every item is checked by hand.
- **Unsure items are left out** of the schema file until confirmed; say so in its header.
- **Conditions of the facts**: `today` (the date checked) and `locale` (the language they hold
  for). The persona used by the task must run in the same locale (`locale:` in the persona), or
  the trial measures a different site.
- **Evidence**: screenshots or other files next to the notes, named in `how_i_know`.
- **Reference solution**: for at least one task, a hand-made output that should score ~100 %. If it
  doesn't, the grader or the ground truth is wrong, not the agent.

## 6. Live sites (a production system we don't own)

Before the first trial, all of these, in the portal config (`portals/<id>/portal.yaml`) and
persona:

- `max_action_class: read` on the portal environment **and** the persona (only look).
- `allowed_domains` lists exact hosts, not a whole domain; small `max_states` / `max_steps` /
  `max_run_time_minutes`; `requests_per_second: 1`; a user agent that says what it is.
- `robots: respect`; read robots.txt yourself and note what it disallows on the task's path.
- Denylist login, account creation and edit paths.
- The owner signs `compliance` (`robots_checked_on`, `terms_reviewed_on`, `terms_reviewed_by`).
  Preflight no longer enforces it; it is our gate, and Claude never signs it on the owner's behalf
  without being told to.
- Run `--crawler-only` unless the BA is part of the question.

## 7. Graders

- **Three groups**: goal (score against ground truth), guardrails (must stay 0, listed in
  `HARD_ZERO_CHECKS`, block promotion), diagnostics (explain why a score moved).
- **Code-based first.** Add a model-based grader only for a category where you showed code gives
  wrong answers, and trust it only after it agrees with human labels on a held-out set; keep those
  labels in `graders/human/`. One narrow yes/no question per judge call, with a reason.
- Grade the **outcome** (what the store holds at the end), not the path the agent took, except a
  trace's steps, which are the outcome of a trace task.
- A grader with an empty denominator scores 0, never 100 %.

### Adding a judge (model-based grader, spec 017)

The mechanism lives in `apps/crawler/evals/graders/model/`; only the judge is new:

1. `graders/model/judges/<id>.ts` exporting `defineJudge({ id, category, question, pass, fail,
   examples, selectPairs })` (`specs/017-llm-judge/contracts/judge-definition.md`), and one line
   in `JUDGES` (`judges/index.ts`).
2. `pnpm eval:draw <id> --portal <p> --store <path>` → `graders/human/labels/<id>.yaml`; the
   owner labels it (pass/fail + one sentence), adding `hand-made` pairs for tricky cases.
3. `pnpm eval:calibrate <id> --split dev`, read every disagreement, refine the wording or a label,
   repeat; then once `--split test`. Calibrated = TPR and TNR ≥ 0.9 on `test` for the current
   wording (changing the wording or model makes it uncalibrated again).

Rules: one narrow yes/no question per call about one (item, record) pair; `fail` covers "not
clearly stated"; `selectPairs` is pure, deterministic, and looser than the code matcher if the
judge is to find false fails (whole words, not substrings); examples come from `dev` labels,
never `test`; the judge sees only the item and the record. Judges run in `pnpm eval` (off with
`--no-judges`), their scores sit in `judges`, never in `scores`, and never gate a baseline. Never
in `pnpm test`; their pure parts are unit-tested on saved CLI outputs.
- Graders that need an agent that did not run (the BA under `--crawler-only`) are skipped, not
  scored 0.

## 8. Trials, conditions and baselines

- Each trial runs in a **fresh, isolated workspace**; never the developer's store unless asked
  (`--in-place`).
- **No result on an outage**: an API or rate-limit error aborts the trial; running out of the
  eval's own budget is a fair end and is recorded.
- **Conditions**: record whatever makes trials comparable in `conditions` (session timeout,
  `crawler_only`, …) and warn when the baseline's differ. Old results without a condition mean
  the default.
- `--dry-run` exercises everything up to the first paid call (workspace, vault, ground truth).
- Run several trials before trusting a score; set `tolerances.json` from their spread.
- **Baseline** = a copy of the trial chosen as the standard, not the latest one. `--promote` does
  it for the trial just run (refused when the crawler did not complete or a guardrail is above 0;
  the old baseline moves to `history/`). Promoting an earlier trial: copy its file to
  `baseline.json` after the same checks; the trial file stays.
- **Re-baseline** when the conditions change (prompt, agent definition, persona, locale, model,
  crawler-only): delete or replace the baseline instead of comparing across the change.
- Result files and baselines are committed; each records the `git_sha` it ran on, so commit before
  a trial.
- The paid trial is never part of `pnpm test`; its pure parts are unit-tested in `evals/tests/`
  (queries against the migrated schema, config parsing, cost, graders, results and promotion).

## 9. Wiring and checks

- Script entry `"eval": "tsx evals/index.ts"` in the app's `package.json`; add `evals` to the
  root `tsconfig.json` `include` and `evals/tests/**/*.test.ts` to a fast Vitest project.
- Run the app's commit checks (CLAUDE.md "Verification Before Commit"); a change to a shared
  schema in `gates` also needs the slow tier.
- When restructuring an existing eval: lock behaviour first (G-29): save the `--dry-run` and
  usage outputs, and compare grader output old vs new on a stored trial's data.
- Update the app's `CLAUDE.md` layout line, the eval's `README.md` and the glossary entries
  (`eval`, `eval-task`, `eval-trial`, `grader`) in the same commit (G-30).

## 10. New eval checklist

1. On paper first (`learning/10-writing-evals.md` steps 0–4): success in words, tasks, ground
   truth, graders.
2. Copy the layout above; fill `tasks.ts`, the ground truth and `prompts.ts` first.
3. Harness: workspace, environment server (or none for a live site), agent runner, cost.
4. Code-based graders, then `grade.ts`; model or human graders only when §7 allows.
5. `results.ts` on the gates schema; tests; `--dry-run`.
6. Validate the eval itself: the reference solution scores ~100 %, a bad output scores low.
7. A few real trials; set tolerances; promote a typical one as the baseline.

Versions

The database has no other skills named create-eval.

Database snapshot: 13.11.2026 · package version a7b61e7e2c83 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100agent-eval-tests
    Use when writing, editing, or reviewing evalite-scored agent evals in packages/core/compute/assistant-evals/src/evals. Use when creating new eval files, adding deterministic assertions or an LLM-judge scorer, or fixing a failing/mis-scoring eval.
  • 95Evals
    Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to…
  • 100eval-harness
    Build and run evaluation suites for AI features in the user's own application: prompts, LLM calls, RAG pipelines, agents. Covers capability and regression evals, graders (exact match, regex, code execution, LLM judge with structured output, human) and metrics (pass@k, pass^k,…
  • 100harness-eval
    Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or…
  • 95Evals
    Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to…
  • 84validate
    Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.