eval
/eval
Use to build and run evals for the AI agent (or LLM app, chatbot or pipeline) in this repository - the harness that runs the real agent (first time), hard realistic tests from risks, requirements, failed runs or traces, harder or combined tests, benchmarks, and shared evaluators ("create evals for this agent", "turn this failure into a test", "set up the harness", "bring in this benchmark"); or to run them and report where the agent stands without changing it ("run the agent evals", "get a baseline", "is the agent getting worse"). Also to fix a test or evaluator that errors or cannot fail, or
- Author
- relai-ai ↗
- Repository
- agent-eval-loop ↗
- Licence
- Apache-2.0
- Last repository change
- 12.11.2026
Assessment and security
- Files in this package
- 11
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Use to build and run evals for the AI agent (or LLM app, chatbot or pipeline) in this repository - the harness that runs the real agent (first time), hard realistic tests from risks, requirements, failed runs or traces, harder or combined tests, benchmarks, and shared evaluators ("create evals for this agent", "turn this failure into a test", "set up the harness", "bring in this benchmark"); or to run them and report where the agent stands without changing it ("run the agent evals", "get a baseline", "is the agent getting worse"). Also to fix a test or evaluator that errors or cannot fail, or
Triggers from the description
Original author description
Use to build and run evals for the AI agent (or LLM app, chatbot or pipeline) in this repository - the harness that runs the real agent (first time), hard realistic tests from risks, requirements, failed runs or traces, harder or combined tests, benchmarks, and shared evaluators ("create evals for this agent", "turn this failure into a test", "set up the harness", "bring in this benchmark"); or to run them and report where the agent stands without changing it ("run the agent evals", "get a baseline", "is the agent getting worse"). Also to fix a test or evaluator that errors or cannot fail, or
How it works
- Pick the mode
- Done when (build)
- Rules
- Ask the user
- Approve the scenarios
- Preview the graders before freezing
- Approach
SKILL.md preview
---
name: eval
description: Use to build and run evals for the AI agent (or LLM app, chatbot or pipeline) in this repository - the harness that runs the real agent (first time), hard realistic tests from risks, requirements, failed runs or traces, harder or combined tests, benchmarks, and shared evaluators ("create evals for this agent", "turn this failure into a test", "set up the harness", "bring in this benchmark"); or to run them and report where the agent stands without changing it ("run the agent evals", "get a baseline", "is the agent getting worse"). Also to fix a test or evaluator that errors or cannot fail, or to mock a dependency trials cannot reach. For agent-neutral coding tasks from sessions or PRs, use the coding-evals skill, including for a custom coding agent developed in this repository.
---
# Build and run evals
Inspect existing tests, runners and results first. Read the relevant terms and guarantees in
`relai/README.md`, or the bundled [contract](../start/template/README.md#terms) when absent.
Use the [start skill](../start/SKILL.md)
when RELAI's managed artifacts are needed, not to replace working evaluation infrastructure.
Choose the work needed for the request under the [shared scope guidance](../start/template/README.md).
The methods below are starting points, not a required full journey.
For unsuitable or failing supplied code, use the shared
[adaptation guidance](../start/template/README.md#adapting-supplied-implementations).
When building or running evals, the goal is that each important behavior has a test that fails
when the agent lacks it and passes when it has it, runs the real agent under relevant operating
conditions, and was reviewed and frozen before its first run;
and the user has honest numbers on how the agent does on those tests.
Across Build and Run, maintain knowledge for future usefulness under the
[shared policy](../start/template/README.md#maintain-useful-knowledge), respecting read-only work.
Useful discoveries can update existing records without reproducing a wiki or board checklist.
## Pick the mode
- **Build** - implement new or harder tests, with scenario approval and grader preview below.
Add a harness on first use when the selected tests need one.
- **Run** - run selected frozen tests, score them and report where the agent stands
([measure](references/measure.md); its *Done when* applies). On a benchmark, follow
[benchmarks](references/benchmarks.md). To compare runs before and after a change, use the `improve` skill.
## Done when (build)
The outcomes below apply to the selected tests. The listed paths and commands are the
RELAI-managed implementation; a working project suite can meet the same guarantees through its
own task definitions, grading checks, reviewed test version, run records and reporting. Record
the guarantees they provide: faithful execution, graders checked against meaningful pass/fail
examples, a recoverable reviewed test version before real trials, and traceable results. RELAI
filenames are not required.
- **Harness ready** (first time only, [harness](references/harness.md); not needed when the only tests are
a wrapped benchmark, whose own runner runs the agent).
- **Scenarios approved:** the user, or an independent reviewer when the approval is delegated
(by an agreement or a recorded choice in `decisions.md`),
approved the scenario table below.
- Each test is a task under `relai/tests/<id>/` (or a benchmark row) meeting
`relai/docs/files.md`, with a `README.md` stating behavior, scenario, risk probed and each
evaluator; `check_task.py` passes, plus `--verifier` for Harbor or synthetic
passing/failing/empty grading checks for a native runner; with API judges, every judge is shown
able to fail (`relai/bin/judge.py --self-test`, or a synthetic violating trajectory for a
prohibition judge).
- **Graders previewed:** the user, or the delegated independent reviewer, approved the preview below.
- A fresh test review ([review-test](references/review-test.md)) approved each test; `freeze.py` ran before its
first real trial; one real trial per test scored every evaluator without an error.
- You showed the results (representative development failures with relevant trajectory evidence;
protected evidence only through permitted assessment summaries), added and asked judge checks
for each new judge evaluator using the protected-evidence boundary in
[measure](references/measure.md#judge-checks-does-the-judge-score-the-way-the-user-would) and, when
`[scoring].pass_mark` is unset, the pass mark ([measure](references/measure.md)), recorded each probe on
its risk page when useful, and updated relevant knowledge or state for future work.
## Rules
- One test per behavior. Extend an existing test rather than near-duplicating it; a frozen test
changes only under the *Frozen tests* rule.
- Hard but realistic: real data instances and the ambiguity and competing cues that make the
behavior hard. When using a simulated user, its instructions hold only that user's goal and knowledge.
Never put the answer, a diagnosis or a hint in the agent's input.
- Exact criteria are code, meaning is a judge, each criterion its own evaluator; expected values
live in `tests/params.json`, not in evaluator code ([test-design](references/test-design.md)).
- **Both directions.** For any behavior an improvement could overcorrect (escalating, refusing,
asking, searching, calling a tool), also test a case where doing it is right, so an agent
that never does it cannot pass.
- **Split.** Development is the default; held-out tests are optional (`relai/README.md`, Terms).
Recommend one only when comparable cases and task distribution make it meaningful, not from the
count alone: many unique, issue-specific tests are not a useful random split.
- A failure caused by the trial itself (a refused action, a broken harness, a killed container)
is inconclusive, not evidence about the agent; fix the probe.
- **Where tests come from**, in this order: runs the user's feedback judged wrong (replay the run,
[test-design](references/test-design.md)); admitted risks, highest first; coverage gaps and co-occurring
weak signals from a triage (`relai/work/discover/<ingest-id>/lanes.json`); requirements no test
covers; and harder or combined versions of passing tests ([hardening](references/hardening.md), always as
new tests).
- Specialized cases: a benchmark ([benchmarks](references/benchmarks.md)), a criterion shared across tests or
an evaluator to repair ([shared-evaluators](references/shared-evaluators.md)), a dependency trials cannot reach or
must not call ([mocks](references/mocks.md)).
## Ask the user
The user is asked at up to three points in a build: once at the start, at scenario approval and
at the grader preview (or once for both, when the user chose one sitting below). Skip any
question that the request, `relai/decisions.md`, the wiki or an agreement already answers. At the
start, ask together only what is still open:
- **Where to start**, only when the wiki has no open risks and the request names no behavior,
run or benchmark: run the `discover` skill first (recommend this, more strongly when traces of
real runs exist), build from risks read in the code now ([test-design](references/test-design.md),
*Hard cases*), or test one behavior the user names.
- **What the results will decide**, when not evident: catching regressions before merge, gating
a release, comparing models or prompts, or seeing where the agent stands. It sets the split
recommendation, attempts and spend.
- On first use, the harness questions still open ([harness](references/harness.md), *Ask the user*).
- **How involved to be**, when nothing says
([delegated approvals](../start/template/README.md#working-with-the-user)): approve the
scenarios and grader preview yourself (recommended for the first build), or delegate both to a
fresh independent review and see the results with the build's report. For a build of one or two
tests, the user may also choose one sitting: build first (building runs no agent), then show the
scenario with its grader preview together. Record the choice in `decisions.md`.
Name any missing credential variable and the project's credential mechanism (or `relai/.env`)
as an instruction, not a question. Reuse existing setup and continue work that does not need it.
## Approve the scenarios
Unless the approval is delegated or the user chose one sitting, wait for a yes before building. Write each proposed
test's full design to `relai/records/eval/<date>-scenarios.md` ([test-design](references/test-design.md),
*Proposing a test*). In chat, show one compact table, in the order you recommend building: the
behavior, its source, the scenario in one line naming the real data and the trigger that makes
it hard, what failing looks like, and its opposite-direction partner, if any. Below it, give the
cost and time of one trial per test plus judging, in total; a yes approves that as this build's
budget, and you ask again only before exceeding it. Ask which tests to build: all, only
the first few, or change the list.
## Preview the graders before freezing
Freezing stops anyone tuning a test to the agent's results, so preview without running the agent:
score each test's evaluators on synthetic trajectories (a passing one, a failing one that commits
the risk's observable failure, an empty reply) and on any real recorded traces that fit the
scenario. Write every test's full table (evaluator × case → score and the first line of
feedback) to `relai/records/eval/<date>-grader-preview.md`. Size chat previews by the shared
[decision-effort rule](../start/template/README.md#working-with-the-user), reviewing manageable
subsets when needed while keeping the complete artifact and pending decisions visible. For the
subset being approved, show every judge evaluator's rows and every score that differs from what
its case should get (a failing case not scored low,
a passing one below full marks, a missing required deliverable above the minimum; see
[test-design](references/test-design.md#rules), *Missing deliverables and prohibitions*). For a
pure prohibition, preview an actual violation to show that check can fail.
Summarize the rest per test in
one line, such as "4 code checks scored as expected". Ask whether the current subset needs scoring
changes or is missing anything the user cares about; separate independent choices as the shared
guidance describes. Iterate until the answer is no, then review and freeze. When this decision
is delegated, the fresh test review stands in for the user. Build/freeze only the subset whose required decisions
are resolved; do not treat partial review as approval of the remaining tests.
## Approach
- Start from admitted risks, highest impact first, then requirements no test covers. With no
risks, follow the user's answer to *Where to start*. For each test, write down the trigger,
passing, failing and what would make the probe invalid before building.
- Prefer the agent's real configuration, real data and real user simulator. Grade outcomes from
exported state (database changes, files, mock journals) with code; keep judges for meaning.
Files in this package (11)
- SKILL.md 11 KB
- references/benchmarks.md 9 KB
- references/harbor-notes.md 8 KB
- references/hardening.md 3 KB
- references/harness.md 13 KB
- references/measure.md 8 KB
- references/mocks.md 4 KB
- references/review-harness.md 5 KB
- references/review-test.md 3 KB
- references/shared-evaluators.md 3 KB
- references/test-design.md 8 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · relai-ai/agent-eval-loop | 100 | Apache-2.0 |
| jisuzzzz/haechi | 100 | MIT |
| alirezarezvani/claude-skills | 100 | MIT |
| ai-analyst-lab/ai-analyst | 100 | MIT |
| ai-analyst-lab/ai-analyst-plugin | 100 | MIT |
| prashantxo/codeforge | 100 | MIT |
| pspdfkit-labs/agentic-usability | 100 | Apache-2.0 |
| netanel-abergel/pa-skills | 100 | MIT |
| Kanevry/session-orchestrator | 98 | MIT |
Database snapshot: 13.11.2026 · package version 0fc8f0e3de27 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100evals-harness
Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets". - 90om-run-harness-evals
Run or inspect Open Mercato harness evaluations and monitor registered experiments with the local dashboard. Use for "run harness evals", "inspect experiment results", "compare eval runs", "uruchom ewaluacje harnessu", or "sprawdź wyniki eksperymentu". - 100agent-evaluation-harness
Design an evaluation harness for an LLM agent before shipping it. Use when the user is building or rewriting an agent, deciding ship/no-ship, debugging regressions, or mentions golden sets, eval suites, regression tests, trace-level evals, LLM-as-judge, scoring rubrics, or asks… - 100creating-mcp-agent-evals
Use when adding Langfuse MCP agent evals for a tool family of the Apify MCP server ("create evals for the storage tools"), when eval cases fail and you must decide whether the case, the tool, or its description is at fault, or when eval runs show tool errors in Langfuse traces. - 98m365-agent-evaluator
Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts",… - 90run-evals
Run the julianpalomo.dev evals (one case, the calibration or the full set), read the failures, and propose a fix in the prompt, the tools or the settings with a run that proves it. Use when a change touches the bot, the claims or evals/, when /handoff calls it, or when an eval…