eval-set
/eval-set
Builds a small eval set for AI output: 5 to 20 real examples, what each answer must and must not contain, one failing case, how to grade. Use when relying on AI-written text or classifications, or changing their prompt.
- Author
- SHOTClubhouse ↗
- Repository
- shotos ↗
- Licence
- MIT
- Last repository change
- 10.11.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 10.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Builds a small eval set for AI output: 5 to 20 real examples, what each answer must and must not contain, one failing case, how to grade. Use when relying on AI-written text or classifications, or changing their prompt.
Original author description
Builds a small eval set for AI output: 5 to 20 real examples, what each answer must and must not contain, one failing case, how to grade. Use when relying on AI-written text or classifications, or changing their prompt.
How it works
- Context and handoff
- When not to use this
- Steps
- Output
- Checks
SKILL.md preview
--- name: eval-set description: "Builds a small eval set for AI output: 5 to 20 real examples, what each answer must and must not contain, one failing case, how to grade. Use when relying on AI-written text or classifications, or changing their prompt." --- # Build an eval set For output that varies, "it looked fine when I tried it" is not a test. An eval set is a short list of real inputs with written expectations, so every change to the prompt or the model can be checked against the same cases. **Stage:** Hypothesise (reused in Operate and Track). **Rules:** 2 Agree acceptance first; 3 Read back the real result. ## Context and handoff Outcome: a change to the prompt or model can be judged on the same cases by someone who did not write it. Acceptance: the pass bar is written before the first run. Read the acceptance card and actual recurring failure patterns. Include a normal case, an incomplete case and a boundary case where relevant. Use `example-mapping` to resolve unclear rules. Store the cases and grading conditions beside the work they check. ## When not to use this Skip it when the output is deterministic and ordinary tests already cover it, or for a one-off task where nobody will repeat the run. Do not invent inputs when real ones exist. ## Steps 1. **Name the task and the owner.** One sentence on what the AI produces, and the person who decides what a good answer is. 2. **Collect real inputs.** Take 5 to 20 examples from actual work: past emails, tickets, documents, records. Remove names, contact details and anything private before they are saved in the project. If you only have a few real cases, start with those and say so rather than padding with invented ones. 3. **Cover the spread.** Include easy cases, awkward ones, an input with missing information, an input that tries to steer the AI off its job, and at least one case where the right answer is to refuse, ask a question or say "I don't know". 4. **Write expectations for each case.** For each input list: - `must contain`: facts, points or actions the answer needs, written as short phrases or alternatives separated by `|`. - `must not contain`: things that would make the answer wrong or unsafe, such as an invented figure, a promise, a name, a refund. - `notes`: why this case is here. 5. **Include at least one case that should fail.** Add an input where the current prompt or model is known or expected to go wrong, or a deliberately wrong answer the grader should reject. A set that every version passes is not testing anything. 6. **Store it in the project.** Save as JSON (for graders that read it) or markdown (for reading). Keep it next to the work, for example `evals/<task>.json`. Do not leave it in the chat. 7. **Say how to grade it.** - Automatic first: check `must contain` and `must not contain` with case-insensitive text matching. Say it is crude and can be fooled. - Then a person reads a sample, at least the failures and every case that passed only barely. - Record for each run: date, model and prompt version, cases passed, cases failed, and who read the sample. - Pass bar: state it before the run (for example "every must-not case passes and at least 80% of the rest"). 8. **Keep it alive.** Add every real failure found later as a new case; that is the correction loop. Read each run's failures back against the case, not just the count. Remove cases that no longer describe the work, and say why. ## Output Use [template.md](template.md). Save it with the work, for example `evals/<task>.json` or `evals/<task>.md`. The receipt for a run is one line per run (date, model, prompt version, passed, failed, who read the sample); say plainly if no run has happened yet. ## Checks - Between 5 and 20 cases, drawn from real examples with private details removed. - Each case has must contain, must not contain and a note. - At least one case should fail or refuse. - Grading method and pass bar are written before the first run. - The set lives in the project, not only in the conversation.
Files in this package (3)
- README.md 3 KB
- SKILL.md 4 KB
- template.md 1 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · SHOTClubhouse/shotos | 100 | MIT |
| fullymiddleaged/clawness | 100 | MIT |
Database snapshot: 13.11.2026 · package version 627af7445240 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100agent-eval-harness
Use to test whether an AI agent, prompt, or skill actually works before or after it ships — build a small eval set, score runs, and catch regressions when the prompt or model changes. Trigger when the user asks whether a prompt is good, wants to compare models or versions, sees… - 100llm-eval-builder
Builds a small, repeatable evaluation set and runner for an LLM feature, with deterministic checks, optional rubric grading, and baseline comparison that catches regressions after a prompt or model change. Use when adding evals to an AI feature, checking whether a prompt or… - 100eval-harness
Build an offline evaluation set from real cases before building an enterprise AI agent, then score every agent version with pass^k, escalation precision and recall, cost per correct outcome, and a regression gate. Use when starting an agent project, before changing a prompt or… - 90prompt-eval-debug
Debug and improve any prompt without rewriting blindly. Builds a tiny eval suite (control, edge, capability-boundary cases), diagnoses failures as prompt issue, missing tool or capability, or harness/workflow issue, and proposes the smallest next change. Use for prompt debugger,… - 100add-eval
Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when… - 100build-eval-dataset
Use this to build a good evaluation dataset for an LLM app, the part everyone underestimates. Trigger on "make an eval set", "what should I test my LLM on", "I don't have test data for my prompt", "build a golden dataset", or before setting up evals. A great eval set beats a…