Skip to content
Skills Library
AI & agents · Testing

eval-driven-development

/eval-driven-development

Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.

Licence
CC0-1.0
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

CC0-1.0, otwarta.

What it does and when to use it

Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.

Triggers from the description

how do I improve this promptmy changes keep breaking other thingshow do I know if this is betteriterate on my agent
Original author description

Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.

How it works

  1. The loop
  2. Wire it into the workflow
  3. Why it works for "vibe coding"
  4. Anti-patterns
Original SKILL.md headings · 32 lines, 379 words
SKILL.md preview
---
name: eval-driven-development
description: Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.
license: CC0-1.0
---

# Eval-driven development (TDD for LLMs)

Changing a prompt and eyeballing one output is how LLM apps quietly regress. Eval-driven development flips it: define what "good" means as a small test set, then iterate until you pass it. It is the single habit that separates apps that get more reliable over time from ones that drift.

## The loop

1. **Write the eval first.** Before touching the prompt, collect 10 to 30 real input cases and define what a good output looks like (a reference answer, or a rubric for LLM-as-a-judge). Include the failure cases you already know about.
2. **Run it against the current version.** This is your baseline score. Now you have a number, not a feeling.
3. **Make one change.** New prompt, different model, changed retrieval. One at a time so you know what moved the score.
4. **Re-run the evals.** Kept the score or improved it? Keep the change. Dropped it? Revert. No debate.
5. **Add every new bug as a case.** When something breaks in production, capture that input as a new eval case before you fix it. The suite grows into a regression net.

## Wire it into the workflow

- Keep the eval set in the repo (versioned), next to the prompts it tests.
- Run it in **CI on every PR** that touches prompts, models, or retrieval, with thresholds that fail the build on a regression.
- Pull real cases from **production traces** (this is where observability feeds evals) so the suite reflects reality, not toy inputs.

See the `add-llm-evals` skill for framework setup (promptfoo, DeepEval, Ragas) and how to calibrate an LLM judge.

## Why it works for "vibe coding"

You can move fast and change prompts freely precisely *because* the eval set catches regressions. Without it, every change is a gamble and confidence drops over time. With it, you get the speed of vibes plus a safety net.

## Anti-patterns

- Tuning a prompt against a single example (you overfit to that one case and break others).
- A dataset written after the fact to match current behavior (it can never catch a regression).
- Changing three things at once, then not knowing which helped.
- Evals that live on someone's laptop instead of in CI (they rot immediately).

Versions

RepositoryScoreLicence
This page · ContextJet-ai/awesome-llm-observability100CC0-1.0
andreibesleaga/gabbe100CC-BY-4.0
ishandutta2007/awesome-agent-skills100MIT
jacob-balslev/skill-graph95Apache-2.0

Database snapshot: 13.11.2026 · package version c47653af9af3 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100eval-harness
    Eval-Driven Development (EDD) framework for AI development. Use when implementing AI features, prompt engineering, or LLM integration. Keywords: eval, evaluation, test, AI, LLM, prompt, benchmark, quality, EDD.
  • 100eval
    Eval-driven development for AI features. Triggers on /codeforge:eval or when any LLM-powered feature, prompt, or agent changes. Define pass/fail before implementing.
  • 100eval-prompt
    Design and run a small evaluation for a prompt, system prompt, Agent Skill or LLM feature, and compare versions. Use when the user asks "is this prompt better?", "test my skill", "evaluate the agent", "build an eval", or wants to change a prompt without breaking what works.
  • 90llm-eval-harness
    This skill should be used when the user asks to "evaluate a prompt", "build an eval", "test my LLM feature", "compare models", "will switching to Haiku/Sonnet break quality", "regression test prompts", or before changing a model ID, effort level or prompt in production code.…
  • 100agentic-eval-first-development
    Architect, execute, and iterate on AI evaluations using the Data-Task-Score framework. Treats evals as the modern, quantifiable version of a PRD. Use when the user asks to "build an eval," "improve model quality," "test an agent workflow," "quantify product intuition," "move…
  • 100fai-eval-driven-dev
    Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.