eval
/eval
Advanced: needs a gold-case YAML you supply and a live data connection. Run the held-out gold suite live against the analyst and score it. Trigger on "/eval", "run the eval suite", "score the system", "run the train split", "check the test split", "what's our accuracy on the gold cases". Drives the analyst on each question (does not grade hand-supplied answers). Pairs with /reliability (stability, no key) and /context-compare (two configs).
- Author
- ai-analyst-lab ↗
- Repository
- ai-analyst-plugin ↗
- Licence
- MIT
- Last repository change
- 26.08.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.08.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Advanced: needs a gold-case YAML you supply and a live data connection. Run the held-out gold suite live against the analyst and score it. Trigger on "/eval", "run the eval suite", "score the system", "run the train split", "check the test split", "what's our accuracy on the gold cases". Drives the analyst on each question (does not grade hand-supplied answers). Pairs with /reliability (stability, no key) and /context-compare (two configs).
Triggers from the description
Original author description
Advanced: needs a gold-case YAML you supply and a live data connection. Run the held-out gold suite live against the analyst and score it. Trigger on "/eval", "run the eval suite", "score the system", "run the train split", "check the test split", "what's our accuracy on the gold cases". Drives the analyst on each question (does not grade hand-supplied answers). Pairs with /reliability (stability, no key) and /context-compare (two configs).
How it works
- Purpose
- The gold-case file
- Invocation
- How to run it
- Model comparison
- Notes
SKILL.md preview
---
name: eval
description: >-
Advanced: needs a gold-case YAML you supply and a live data connection. Run the held-out gold
suite live against the analyst and score it. Trigger on "/eval", "run the eval suite", "score
the system", "run the train split", "check the test split", "what's our accuracy on the gold
cases". Drives the analyst on each question (does not grade hand-supplied answers). Pairs with
/reliability (stability, no key) and /context-compare (two configs).
---
# Skill: Eval (live gold-suite runner)
## Purpose
Run the analyst on every question in a held-out gold suite, then score the answers against the
blind gold: accuracy (the analyst's number vs the recomputed gold), query similarity (its SQL vs
the trusted query), and cost/latency. This is the system-level eval: the number that tells you
whether the context you are adding is paying off, and the number a model comparison turns on.
Two honest properties:
- **Blind by construction.** The analyst runs see the **question only**, never the gold SQL or
value. The gold is read only at grading, after the answers are locked.
- **Real, not staged.** Each answer is produced by actually running the analyst now. Nothing is
pre-filled.
## The gold-case file
The suite is a YAML file you author and keep in your working folder (for example
`gold-cases.yaml`). It never lives in shared context, because the analyst must not be able to see
it. Each case is a question you already know the right answer to, paired with the query you trust
and the value it returns:
```yaml
cases:
- id: rev-2025-q4 # short unique id
question: "What was total net revenue in Q4 2025?"
split: train # train (the set you iterate on) or test (held out)
gold_sql: "select sum(net_revenue) from orders where order_date between '2025-10-01' and '2025-12-31'"
gold_value: 4823910.55 # what gold_sql returns; a reference point, recomputed at grading
# tolerance: 0.01 # optional relative tolerance; default 0.005 (0.5%)
```
The easiest way to start: take five queries your team already trusts (month-end numbers you have
reported, dashboard tiles you have verified) and record each as a case with the exact SQL and the
value it produces. Mark three or four of them `train` and keep at least one as `test`. Grading
recomputes the gold by re-running `gold_sql` against the live connection at eval time, so the
suite does not rot as new data arrives; the stored `gold_value` is a sanity reference.
## Invocation
`/eval [train|test|all] [--slice N]`, default split `train`.
- `train`: the working set you iterate on (error-analyze, add context, watch accuracy climb).
Default.
- `test`: the held-out set. Run this ONCE at the end as the honest generalization number. Never
iterate against it.
- `--slice N`: run only the first N cases as a quick spot check. Omit for the full split.
Examples: `/eval train` · `/eval train --slice 3` · `/eval test`
## How to run it
### Step 0: preflight (fail loud)
Verify the live data connection before anything runs: execute a trivial probe query
(`select 1`) through the session's active warehouse connection. If the probe fails, stop and
surface the error clearly; there is NO fallback engine. Do not grade against any other engine
than the one the gold was written for.
### Step 1: get the questions (blind)
Ask for the gold file's path if you do not have it. Load the question set for the requested split
by reading ONLY each case's `question` and `split` fields. Do not read `gold_sql` or `gold_value`
at this stage, so you cannot leak the key. If `--slice N` was given, take the first N.
### Step 2: run the analyst once per question
Launch one **fresh sub-agent per question** with the Task/Agent tool (run them concurrently in
reasonable batches). Each sub-agent gets a fresh context and sees ONLY its question. Time each run
(wall-clock) for latency. Give each sub-agent exactly this brief:
> You are answering one analytics question against the active dataset. Load the normal session
> context first (knowledge-bootstrap: `.knowledge/active.yaml`, then the active dataset's `schema.md`,
> `quirks.md`, and manifest from the local datasets dir). For the metric dictionary and semantic
> context, first resolve the context dir: read `.knowledge/context-source.yaml`; if it exists and
> says `source: git`, clone or pull the repo it names into `.knowledge/.context-cache` (checkout
> the `ref`, default `main`) and use the dataset's directory inside that cache; otherwise use
> `.knowledge/datasets/{active}/`. Read `metrics/index.yaml`, `semantic/`, and the verified queries
> (`verified_queries.yaml`, at the dataset root or under `semantic/` in older layouts) from the
> RESOLVED dir, not from any other copy. Do not read `.knowledge/reliability/` history before
> answering. If the metric you're asked about is defined in the dictionary, use that definition
> exactly. If it is NOT defined, decide for yourself how best to define and measure it. Query the
> real warehouse through the session's active data connection. Answer the question: "<QUESTION>".
> Return ONLY this block:
> `analyst_value: <the single number you'd report, digits only>`
> `analyst_query: <the exact SQL you ran to get it, one line>`
> `definition_source: <"metric dictionary" if you used a defined metric, else "my own choice">`
The undefined-metric cases are the ones that drift and fail at baseline; they flip to pass once
their definitions are added to the metric dictionary. That is the point of the exercise, so do not
hand the sub-agents definitions they don't have in the dictionary.
### Step 3: assemble the per-case results
Build one record per question:
`{"question", "analyst_value", "analyst_query", "latency_ms"}` (add `tokens`/`cost` per case only if
you can measure them honestly; otherwise capture run-level cost in Step 4).
### Step 4: grade against the blind gold + write the run record
Only NOW, after every analyst answer is locked, open the gold file. Grade deterministically by
writing and running a small Python script (never eyeball or estimate the numbers):
- **Accuracy (the suite metric).** For each case, recompute the gold value by running the case's
`gold_sql` against the live connection (recomputing at eval time means the gold cannot rot).
Parse the analyst's number (strip `$`, `%`, commas; expand `k`/`m`/`b` suffixes). Pass if it
matches the recomputed gold within the case's relative tolerance (default 0.5%). Unit-aware
pass: if the gold is a rate (0 < gold <= 1) and the analyst's value is about 100x it, re-check
the analyst value divided by 100; count it as a pass but flag it `unit_adjusted`.
- **Query similarity.** Compare the analyst's SQL to the case's `gold_sql`, both normalized
(lowercase, collapse whitespace, strip trailing semicolon): report `identical` or `changed`,
plus a 0..1 Jaccard token-overlap similarity over the normalized SQL tokens. This measures text
closeness, not semantic equivalence; say so if asked.
- **Aggregate.** `accuracy = passed/total`, `avg_query_similarity`, and total cost/latency when
honestly captured (omit cost if you cannot measure it).
Write a self-describing run record `<run_id>.json` in a runs directory next to the gold suite,
carrying: timestamp, `split`, model, a one-line `changelog` (what changed since last run),
`context_state` (which metrics are currently defined in the resolved context dir's
`metrics/index.yaml`, the list that grows as you add definitions), the aggregate, and per-case
detail (question, gold value, analyst value, passed, unit_adjusted, analyst query, query_diff,
query_similarity, latency_ms).
### Step 5: report
Show the headline from the record: `accuracy = passed/total`, `avg_query_similarity`, and (when
present) `total_cost` / `cost_per_correct` / `avg_latency_ms`. Then do the **error analysis**
yourself: cluster the failures by mode (undefined-metric-drift, fan-out, wrong-filter,
wrong-grain, wrong-source), rank the modes by count, and record the clustering in the run
record. Then:
- Name the **context_state** (how many metrics are defined). The climb is this growing run over run.
- Show the ranked **failure modes**, then **diagnose** them; the diagnosis is the analyst's job,
not the tool's: what does the dominant mode mean, and what is the fix? (Usually the definitional
failures are missing definitions, so add them to the metric dictionary; a fan-out cluster
usually means a join convention is needed.)
- For **train**: "this is your working number; the clustered failures show what to fix. Add the
missing definitions, re-run, watch it climb."
- For **test**: "held-out number. If train climbed but this didn't, you overfit. Don't tune on this."
- Report the score as what this run produced, not a promised or expected figure. Accuracy depends
on how much context has been defined, so numbers differ between setups.
## Model comparison
To compare engines or configurations, run `/eval train` in two sessions, one per configuration,
then put the two run records side by side with `/context-compare`. The cell that matters is
`cost_per_correct`. Run the comparison on **train** so the held-out test stays pristine.
## Notes
- The run record carries the `changelog` and `context_state`, so a sequence of `/eval` run records
reads as a trend line over time: context grows, accuracy climbs.
- Blind discipline: never paste gold SQL or values into a sub-agent's context. If you need to debug
a failure, read the run record's per-case detail (it shows gold next to analyst for the human),
not the sub-agent.
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · ai-analyst-lab/ai-analyst-plugin | 100 | MIT |
| relai-ai/agent-eval-loop | 100 | Apache-2.0 |
| jisuzzzz/haechi | 100 | MIT |
| alirezarezvani/claude-skills | 100 | MIT |
| ai-analyst-lab/ai-analyst | 100 | MIT |
| prashantxo/codeforge | 100 | MIT |
| pspdfkit-labs/agentic-usability | 100 | Apache-2.0 |
| netanel-abergel/pa-skills | 100 | MIT |
| Kanevry/session-orchestrator | 98 | MIT |
Database snapshot: 13.11.2026 · package version e10a6b5197c0 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 84validate
Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample… - 95agent-eval
Run eval fixtures against review agents and grade results. Use this after adding or modifying a review agent, to validate detection accuracy, or when the user says "run the evals", "test the agents", "check for regressions", or "how accurate is the agent". - 100run
Run the agent-config eval suite locally and compare with the stored baseline; explain why a case went red. Use when the user says "run the evals", "run config-drift-checker", "did the Claude Code update break my setup", "why is the eval red", or "compare against baseline". Do… - 100skill-eval
Measure whether a skill actually helps: build a `claude plugin eval` suite, run it with and without the skill under a required spend cap, and read the delta. Use when adding evals to a skill, converting evals.json, re-checking skills after a model upgrade, or deciding if a skill… - 90write-case
Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the… - 100evaluate
Evaluate a Manykind workflow with `mk eval`. Keep a dataset of cases in the repo (seeded from intent.md's Examples), sync it with the agent service, run it as an experiment against a workflow version, and compare experiments. Use when the user asks to test, score, benchmark,…