eval

What it does and when to use it

Eval-driven development for AI features. Triggers on /codeforge:eval or when any LLM-powered feature, prompt, or agent changes. Define pass/fail before implementing.

How it works

1. Workflow
2. Eval spec format
3. Grader types
4. Metrics
5. Report format

---
name: eval
description: Eval-driven development for AI features. Triggers on /codeforge:eval or when any LLM-powered feature, prompt, or agent changes. Define pass/fail before implementing.
allowed-tools: Read, Write, Edit, Bash, Grep, Glob
---

# codeforge:eval

## Workflow

| Phase | Action | Gate |
|-------|--------|------|
| DEFINE | Write eval spec before any code | **STOP: show user, wait for approval** |
| IMPLEMENT | Write code/prompts to pass the spec | n/a |
| GRADE | Run graders, measure pass@k | n/a |
| REPORT | Compare to baseline, flag regressions | Block on any regression |

## Eval spec format

```markdown
## EVAL: <feature-name>
### Capability
- [ ] <what must succeed>
### Regression
- [ ] <existing behavior that must not break>
### Metrics
- Capability: pass@3 > 90%
- Regression: pass^3 = 100%
```
Save at `.claude/evals/<feature-name>.md`.

## Grader types

| Output | Grader | Example |
|--------|--------|---------|
| Deterministic | Code (bash) | `npm test && echo PASS \|\| echo FAIL` |
| Open-ended | Model prompt | Rate 1-3: solves problem / correct format / handles edges |

## Metrics

| Metric | Meaning | Target |
|--------|---------|--------|
| pass@1 | First-attempt success | > 70% |
| pass@3 | At least 1 success in 3 | > 90% |
| pass^3 | All 3 succeed | 100% on critical paths |

## Report format

```
FORGE EVAL — <date>
Feature       Capability    Regression    Status
<name>        X/Y (N%)      X/Y (N%)      PASS|FAIL|REGRESSED
Baseline delta: <vs last run>
```

Versions

Database snapshot: 13.11.2026 · package version 5ab1455d02d2 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

• 100eval-harness
  Eval-Driven Development (EDD) framework for AI development. Use when implementing AI features, prompt engineering, or LLM integration. Keywords: eval, evaluation, test, AI, LLM, prompt, benchmark, quality, EDD.
• 100eval-prompt
  Design and run a small evaluation for a prompt, system prompt, Agent Skill or LLM feature, and compare versions. Use when the user asks "is this prompt better?", "test my skill", "evaluate the agent", "build an eval", or wants to change a prompt without breaking what works.
• 100eval-driven-ai-product
  Use when the feature is AI/LLM-powered (generation, summarization, classification, agents, RAG) — its acceptance is an EVAL, not "it runs." Define the dataset, graders, thresholds, and failure taxonomy before shipping.
• 100eval-driven-development
  Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate…
• 100llm-eval-builder
  Builds a small, repeatable evaluation set and runner for an LLM feature, with deterministic checks, optional rubric grading, and baseline comparison that catches regressions after a prompt or model change. Use when adding evals to an AI feature, checking whether a prompt or…
• 100ai-feature-eval-harness
  Design an eval plan for a product AI feature: success criteria, a held-out labeled dataset, per-criterion grading (code-based, then LLM-based), and a pass threshold. Use when a task ships or changes a feature with non-deterministic model output (classification, extraction,…