Skip to content
Skills Library
AI & agents · Testing

Evals

/Evals

Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner.

Repository
pai ↗
Licence
MIT
Last repository change
09.06.2026

Assessment and security

95/ 100Class: good
Format20 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
38
Scripts
19
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 09.06.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 19 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: Graders/Base.ts, Graders/CodeBased/BinaryTests.ts, Graders/CodeBased/RegexMatch.ts, Graders/CodeBased/StateCheck.ts, Graders/CodeBased/StaticAnalysis.ts, Graders/CodeBased/StringMatch.ts, Graders/CodeBased/ToolCallVerification.ts, Graders/CodeBased/index.ts, Graders/ModelBased/LLMRubric.ts, Graders/ModelBased/NaturalLanguageAssert.ts, Graders/ModelBased/PairwiseComparison.ts, Graders/ModelBased/in

What it does and when to use it

Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner.

Original author description

Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner.

How it works

  1. Customization
  2. 🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)
  3. When to Activate
  4. Core Concepts
  5. Workflow Routing
  6. Quick Reference
  7. Available Graders
  8. Domain Patterns
  9. Task Schema (YAML)
  10. Resource Index
Original SKILL.md headings · 237 lines, 993 words
SKILL.md preview
---
name: Evals
description: Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner.
---

## Customization

**Before executing, check for user customizations at:**
`~/.claude/PAI/USER/SKILLCUSTOMIZATIONS/Evals/`

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.


## 🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

**You MUST send this notification BEFORE doing anything else when this skill is invoked.**

1. **Send voice notification**:
   ```bash
   curl -s -X POST http://localhost:8888/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \
     > /dev/null 2>&1 &
   ```

2. **Output text notification**:
   ```
   Running the **WorkflowName** workflow in the **Evals** skill to ACTION...
   ```

**This is not optional. Execute this curl command immediately upon skill invocation.**

# Evals - AI Agent Evaluation Framework

Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).

**Key differentiator:** Evaluates agent *workflows* (transcripts, tool calls, multi-turn conversations), not just single outputs.

---

## When to Activate

- "run evals", "test this agent", "evaluate", "check quality", "benchmark"
- "regression test", "capability test"
- Compare agent behaviors across changes
- Validate agent workflows before deployment
- Verify ALGORITHM ISC rows
- Create new evaluation tasks from failures

---

## Core Concepts

### Three Grader Types

| Type | Strengths | Weaknesses | Use For |
|------|-----------|------------|---------|
| **Code-based** | Fast, cheap, deterministic, reproducible | Brittle, lacks nuance | Tests, state checks, tool verification |
| **Model-based** | Flexible, captures nuance, scalable | Non-deterministic, expensive | Quality rubrics, assertions, comparisons |
| **Human** | Gold standard, handles subjectivity | Expensive, slow | Calibration, spot checks, A/B testing |

### Evaluation Types

| Type | Pass Target | Purpose |
|------|-------------|---------|
| **Capability** | ~70% | Stretch goals, measuring improvement potential |
| **Regression** | ~99% | Quality gates, detecting backsliding |

### Key Metrics

- **pass@k**: Probability of at least 1 success in k trials (measures capability)
- **pass^k**: Probability all k trials succeed (measures consistency/reliability)

---

## Workflow Routing

| Request Pattern | Route To |
|---|---|
| Run eval, evaluate suite, run tests, benchmark | `Workflows/RunEval.md` |
| Compare models, model comparison, A/B test models | `Workflows/CompareModels.md` |
| Compare prompts, prompt comparison, test prompts | `Workflows/ComparePrompts.md` |
| Create judge, model grader, evaluation judge | `Workflows/CreateJudge.md` |
| Create use case, new eval, test case, create suite | `Workflows/CreateUseCase.md` |
| View results, eval results, scores, pass rate | `Workflows/ViewResults.md` |

### CLI Quick Reference

| Trigger | Tool |
|---------|------|
| Run suite | `Tools/AlgorithmBridge.ts` |
| Log failure | `Tools/FailureToTask.ts log` |
| Convert failures | `Tools/FailureToTask.ts convert-all` |
| Create suite | `Tools/SuiteManager.ts create` |
| Check saturation | `Tools/SuiteManager.ts check-saturation` |

---

## Quick Reference

### CLI Commands

```bash
# Run an eval suite
bun run ~/.claude/skills/Utilities/Evals/Tools/AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ~/.claude/skills/Utilities/Evals/Tools/FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ~/.claude/skills/Utilities/Evals/Tools/FailureToTask.ts convert-all

# Manage suites
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts list
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts check-saturation <name>
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts graduate <name>
```

### ALGORITHM Integration

Evals is a verification method for THE ALGORITHM ISC rows:

```bash
# Run eval and update ISC row
bun run ~/.claude/skills/Utilities/Evals/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u
```

ISC rows can specify eval verification:
```
| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |
```

---

## Available Graders

### Code-Based (Fast, Deterministic)

| Grader | Use Case |
|--------|----------|
| `string_match` | Exact substring matching |
| `regex_match` | Pattern matching |
| `binary_tests` | Run test files |
| `static_analysis` | Lint, type-check, security scan |
| `state_check` | Verify system state after execution |
| `tool_calls` | Verify specific tools were called |

### Model-Based (Nuanced)

| Grader | Use Case |
|--------|----------|
| `llm_rubric` | Score against detailed rubric |
| `natural_language_assert` | Check assertions are true |
| `pairwise_comparison` | Compare to reference with position swap |

---

## Domain Patterns

Pre-configured grader stacks for common agent types:

| Domain | Primary Graders |
|--------|-----------------|
| `coding` | binary_tests + static_analysis + tool_calls + llm_rubric |
| `conversational` | llm_rubric + natural_language_assert + state_check |
| `research` | llm_rubric + natural_language_assert + tool_calls |
| `computer_use` | state_check + tool_calls + llm_rubric |

See `Data/DomainPatterns.yaml` for full configurations.

---

## Task Schema (YAML)

```yaml
task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/security_review.md

  trials: 3
  pass_threshold: 0.75
```

---

## Resource Index

| Resource | Purpose |
|----------|---------|
| `Types/index.ts` | Core type definitions |
| `Graders/CodeBased/` | Deterministic graders |
| `Graders/ModelBased/` | LLM-powered graders |
| `Tools/TranscriptCapture.ts` | Capture agent trajectories |
| `Tools/TrialRunner.ts` | Multi-trial execution with pass@k |
| `Tools/SuiteManager.ts` | Suite management and saturation |
| `Tools/FailureToTask.ts` | Convert failures to test tasks |
| `Tools/AlgorithmBridge.ts` | ALGORITHM integration |
| `Data/DomainPatterns.yaml` | Domain-specific grader configs |

---

## Key Principles (from Anthropic)

1. **Start with 20-50 real failures** - Don't overthink, capture what actually broke
2. **Unambiguous tasks** - Two experts should reach identical verdicts
3. **Balanced problem sets** - Test both "should do" AND "should NOT do"
4. **Grade outputs, not paths** - Don't penalize valid creative solutions
5. **Calibrate LLM judges** - Against human expert judgment
6. **Check transcripts regularly** - Verify graders work correctly
7. **Monitor saturation** - Graduate to regression when hitting 95%+
8. **Build infrastructure early** - Evals shape how quickly you can adopt new models

---

## Related

- **ALGORITHM**: Evals is a verification method
- **Science**: Evals implements scientific method
- **Browser**: For visual verification graders
Files in this package (38)
  • BestPractices.md 2 KB
  • CLIReference.md 2 KB
  • PROJECT.md 21 KB
  • SKILL.md 8 KB
  • ScienceMapping.md 2 KB
  • ScorerTypes.md 2 KB
  • TemplateIntegration.md 2 KB
  • Data/DomainPatterns.yaml 5 KB
  • Graders/Base.ts 3 KB
  • Graders/index.ts 1 KB
  • Graders/CodeBased/BinaryTests.ts 2 KB
  • Graders/CodeBased/RegexMatch.ts 2 KB
  • Graders/CodeBased/StateCheck.ts 5 KB
  • Graders/CodeBased/StaticAnalysis.ts 3 KB
  • Graders/CodeBased/StringMatch.ts 2 KB
  • Graders/CodeBased/ToolCallVerification.ts 4 KB
  • Graders/CodeBased/index.ts 1 KB
  • Graders/ModelBased/LLMRubric.ts 5 KB
  • Graders/ModelBased/NaturalLanguageAssert.ts 4 KB
  • Graders/ModelBased/PairwiseComparison.ts 5 KB
  • Graders/ModelBased/index.ts 1 KB
  • Suites/Regression/core-behaviors.yaml 1 KB
  • Tools/AlgorithmBridge.ts 7 KB
  • Tools/FailureToTask.ts 10 KB
  • Tools/SuiteManager.ts 11 KB
  • Tools/TranscriptCapture.ts 6 KB
  • Tools/TrialRunner.ts 8 KB
  • Types/index.ts 9 KB
  • UseCases/Regression/task_file_targeting_basic.yaml 1 KB
  • UseCases/Regression/task_no_hallucinated_paths.yaml 1 KB
  • UseCases/Regression/task_tool_sequence_read_before_edit.yaml 1 KB
  • UseCases/Regression/task_verification_before_done.yaml 1 KB
  • Workflows/CompareModels.md 7 KB
  • Workflows/ComparePrompts.md 9 KB
  • Workflows/CreateJudge.md 5 KB
  • Workflows/CreateUseCase.md 6 KB
  • Workflows/RunEval.md 2 KB
  • Workflows/ViewResults.md 7 KB

Versions

RepositoryScoreLicence
This page · virtualian/pai95MIT
emplio-app/emplio100AGPL-3.0
renzrollon/interlock100MIT
soummyaanon/engineering-os100MIT
chrishuffman5/domain-expert100MIT
houseofmvps/ultraship100MIT
camronh/EZVals95MIT
virtualian/pai95MIT
virtualian/pai95MIT

Identical content appears in 3 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version 3959b04a9c23 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 2 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100eval-harness
    Build and run evaluation suites for AI features in the user's own application: prompts, LLM calls, RAG pipelines, agents. Covers capability and regression evals, graders (exact match, regex, code execution, LLM judge with structured output, human) and metrics (pass@k, pass^k,…
  • 90eval-writer
    Create new eval suites for the deepagentsjs monorepo. Handles dataset design, test case scaffolding, scoring logic, vitest configuration, and LangSmith integration. Use when the user asks to: (1) create an eval, (2) write an evaluation, (3) add a benchmark, (4) build an eval…
  • 100add-eval
    Add or edit a plugin eval case under plugin/evals (claude plugin eval) — prompt, graders, MCP mocks — and run, compare and re-baseline the suite. Use when writing a new eval, changing a grader or mock, checking a skill/agent/tool-description change for regressions, or when…
  • 100bmad-eval-runner
    Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs.
  • 100llm-evals
    Builds evaluation suites for LLM features and agents - error analysis on real outputs, pass criteria, code-based checks, calibrated LLM-as-judge graders, dataset splits, CI regression runs, and noise-aware comparisons with cost and latency. Use when testing prompts or models,…
  • 95agent-eval
    Run eval fixtures against review agents and grade results. Use this after adding or modifying a review agent, to validate detection accuracy, or when the user says "run the evals", "test the agents", "check for regressions", or "how accurate is the agent".

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.