Skip to content
Skills Library
AI & agents · Testing

agent-evaluation

/agent-evaluation

Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures. Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".

Licence
MIT
Last repository change
20.07.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 20.07.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures. Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".

Triggers from the description

agent evalbenchmark agenttool accuracyagent regression
Original author description

Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures. Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".

How it works

  1. What to measure
  2. Workflow
  3. Automated checks
  4. Human rubric (when needed)
  5. Anti-patterns
  6. Output
Original SKILL.md headings · 42 lines, 225 words
SKILL.md preview
---
name: agent-evaluation
description: >
  Evaluate LLM agents and tool-using workflows—task success, tool accuracy,
  latency/cost, safety, and regression suites. Use when shipping agent features,
  comparing prompts/models, or debugging agent failures.
  Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".
---

# Agent evaluation

## What to measure

| Dimension | Examples |
|-----------|----------|
| Task success | End state matches spec (binary or rubric) |
| Tool use | Correct tool, valid args, no spurious calls |
| Safety | No policy violations, no secret leakage |
| Efficiency | Tokens, latency, tool call count |
| Stability | Same input -> consistent outcome across runs |

## Workflow

1. **Define tasks** — realistic user intents with clear pass/fail or scored rubric.
2. **Build dataset** — golden set + edge cases (errors, ambiguous input, empty context).
3. **Run baseline** — fixed model/settings; log traces (inputs, tools, outputs).
4. **Score** — automated checks first; human review for ambiguous cases.
5. **Compare** — A/B prompts, models, or tool schemas; report deltas with confidence notes.
6. **Gate** — block release on regression in must-pass tasks.

## Automated checks

- Schema validation on tool arguments.
- Assert final answer contains required fields or avoids forbidden content.
- Snapshot tests for deterministic sub-steps where possible.

## Human rubric (when needed)

Score 1-5 on: correctness, completeness, tone, safety. Document disagreements.

## Anti-patterns

- Eval only on cherry-picked happy paths.
- Changing task and model simultaneously without isolation.
- No trace logs when debugging tool failures.

## Output

Summary table: variant | success rate | avg tools | avg latency | notes.

Versions

RepositoryScoreLicence
This page · charlieviettq/awesome-agent-skill100MIT
FrancoStino/opencode-skills-collection100MIT
helx0071/skillsmo100MIT
alebgl77/claude-inc100MIT
guia-matthieu/clawfu-skills100MIT
seb1n/awesome-ai-agent-skills100MIT
viktorbezdek/skillstack100MIT
mlflow/skills95Apache-2.0
NeoLabHQ/context-engineering-kit91GPL-3.0

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version c0429053f4d0 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100agent-evaluation-and-guardrails
    Guides building evaluation harnesses, regression test suites, and runtime guardrails for LLM agents. Use when a user asks to "evaluate this agent," "write test cases for a prompt change," "set up an eval harness," "add guardrails to prevent unsafe output," "detect prompt…
  • 100llm-evals
    Builds evaluation suites for LLM features and agents - error analysis on real outputs, pass criteria, code-based checks, calibrated LLM-as-judge graders, dataset splits, CI regression runs, and noise-aware comparisons with cost and latency. Use when testing prompts or models,…
  • 100agent-harness
    Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.
  • 100agent-evaluation-harness
    Design an evaluation harness for an LLM agent before shipping it. Use when the user is building or rewriting an agent, deciding ship/no-ship, debugging regressions, or mentions golden sets, eval suites, regression tests, trace-level evals, LLM-as-judge, scoring rubrics, or asks…
  • 100evaluate-agent
    Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions…
  • 90skill-evaluation-workbench
    Use when designing, running, debugging, or hardening deterministic eval suites for agent skills, prompts, tool workflows, or MCP-backed cases.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.