eval-driven-development
/eval-driven-development
Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.
- Author
- ContextJet-ai ↗
- Repository
- awesome-llm-observability ↗
- Licence
- CC0-1.0
- Last repository change
- 26.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.
CC0-1.0, otwarta.
What it does and when to use it
Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.
Triggers from the description
Original author description
Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs.
How it works
- The loop
- Wire it into the workflow
- Why it works for "vibe coding"
- Anti-patterns
SKILL.md preview
--- name: eval-driven-development description: Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG change. This is test-driven development for LLMs. license: CC0-1.0 --- # Eval-driven development (TDD for LLMs) Changing a prompt and eyeballing one output is how LLM apps quietly regress. Eval-driven development flips it: define what "good" means as a small test set, then iterate until you pass it. It is the single habit that separates apps that get more reliable over time from ones that drift. ## The loop 1. **Write the eval first.** Before touching the prompt, collect 10 to 30 real input cases and define what a good output looks like (a reference answer, or a rubric for LLM-as-a-judge). Include the failure cases you already know about. 2. **Run it against the current version.** This is your baseline score. Now you have a number, not a feeling. 3. **Make one change.** New prompt, different model, changed retrieval. One at a time so you know what moved the score. 4. **Re-run the evals.** Kept the score or improved it? Keep the change. Dropped it? Revert. No debate. 5. **Add every new bug as a case.** When something breaks in production, capture that input as a new eval case before you fix it. The suite grows into a regression net. ## Wire it into the workflow - Keep the eval set in the repo (versioned), next to the prompts it tests. - Run it in **CI on every PR** that touches prompts, models, or retrieval, with thresholds that fail the build on a regression. - Pull real cases from **production traces** (this is where observability feeds evals) so the suite reflects reality, not toy inputs. See the `add-llm-evals` skill for framework setup (promptfoo, DeepEval, Ragas) and how to calibrate an LLM judge. ## Why it works for "vibe coding" You can move fast and change prompts freely precisely *because* the eval set catches regressions. Without it, every change is a gamble and confidence drops over time. With it, you get the speed of vibes plus a safety net. ## Anti-patterns - Tuning a prompt against a single example (you overfit to that one case and break others). - A dataset written after the fact to match current behavior (it can never catch a regression). - Changing three things at once, then not knowing which helped. - Evals that live on someone's laptop instead of in CI (they rot immediately).
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · ContextJet-ai/awesome-llm-observability | 100 | CC0-1.0 |
| andreibesleaga/gabbe | 100 | CC-BY-4.0 |
| ishandutta2007/awesome-agent-skills | 100 | MIT |
| jacob-balslev/skill-graph | 95 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version c47653af9af3 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100eval-harness
Eval-Driven Development (EDD) framework for AI development. Use when implementing AI features, prompt engineering, or LLM integration. Keywords: eval, evaluation, test, AI, LLM, prompt, benchmark, quality, EDD. - 100eval
Eval-driven development for AI features. Triggers on /codeforge:eval or when any LLM-powered feature, prompt, or agent changes. Define pass/fail before implementing. - 100eval-prompt
Design and run a small evaluation for a prompt, system prompt, Agent Skill or LLM feature, and compare versions. Use when the user asks "is this prompt better?", "test my skill", "evaluate the agent", "build an eval", or wants to change a prompt without breaking what works. - 90llm-eval-harness
This skill should be used when the user asks to "evaluate a prompt", "build an eval", "test my LLM feature", "compare models", "will switching to Haiku/Sonnet break quality", "regression test prompts", or before changing a model ID, effort level or prompt in production code.… - 100agentic-eval-first-development
Architect, execute, and iterate on AI evaluations using the Data-Task-Score framework. Treats evals as the modern, quantifiable version of a PRD. Use when the user asks to "build an eval," "improve model quality," "test an agent workflow," "quantify product intuition," "move… - 100fai-eval-driven-dev
Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says…