Skip to content
Skills Library
AI & agents · Testing

agent-evaluation

/agent-evaluation

Designs and runs reproducible evaluations for AI agents, prompts, tools, skills, and model-backed workflows using realistic datasets, isolated baselines, objective assertions, rubric grading, trajectory analysis, cost/latency tracking, and regression comparison. Use when measuring agent quality, optimizing skill triggering, comparing prompts or models, or gating an AI feature release. Not for ordinary deterministic unit tests.

Repository
godmode ↗
Licence
MIT
Last repository change
26.08.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.08.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Designs and runs reproducible evaluations for AI agents, prompts, tools, skills, and model-backed workflows using realistic datasets, isolated baselines, objective assertions, rubric grading, trajectory analysis, cost/latency tracking, and regression comparison. Use when measuring agent quality, optimizing skill triggering, comparing prompts or models, or gating an AI feature release. Not for ordinary deterministic unit tests.

Original author description

Designs and runs reproducible evaluations for AI agents, prompts, tools, skills, and model-backed workflows using realistic datasets, isolated baselines, objective assertions, rubric grading, trajectory analysis, cost/latency tracking, and regression comparison. Use when measuring agent quality, optimizing skill triggering, comparing prompts or models, or gating an AI feature release. Not for ordinary deterministic unit tests.

How it works

  1. Define the evaluation contract
  2. Run isolated comparisons
  3. Analyze and iterate
  4. Completion condition
Original SKILL.md headings · 48 lines, 268 words
SKILL.md preview
---
name: agent-evaluation
description: >-
  Designs and runs reproducible evaluations for AI agents, prompts, tools,
  skills, and model-backed workflows using realistic datasets, isolated
  baselines, objective assertions, rubric grading, trajectory analysis,
  cost/latency tracking, and regression comparison. Use when measuring agent
  quality, optimizing skill triggering, comparing prompts or models, or gating
  an AI feature release. Not for ordinary deterministic unit tests.
---

# Agent Evaluation

Build a quality flywheel that can distinguish a real improvement from a lucky
run.

## Define the evaluation contract

Name the target behavior, users, risks, baseline, candidate, environment,
stochastic settings, and decision threshold. Start with a few realistic cases,
including a boundary or failure case. Split trigger-query optimization into a
fixed training set and held-out validation set.

Use [eval-schema.md](references/eval-schema.md) for cases, assertions, timing,
and result records.

## Run isolated comparisons

1. Snapshot the baseline before changing the candidate.
2. Run baseline and candidate on identical inputs in fresh contexts with no
   leaked expected answer or previous trace.
3. Capture final artifacts, public transcript/tool summaries, duration, token
   or request cost, and failures.
4. Grade deterministic assertions first; use a blinded rubric or human review
   for qualities that cannot be measured mechanically.
5. Repeat stochastic cases enough to expose variance. Do not hide flakiness by
   dropping inconvenient runs.

Measure task success, instruction adherence, tool selection and arguments,
trajectory efficiency, grounding, safety, output quality, latency, and cost only
when relevant. A single aggregate score must not hide a release-blocking metric.

## Analyze and iterate

Cluster repeated failures by cause, change one owning layer, rerun the affected
cases, then run the regression set. Compare candidate against baseline and
reject improvements that regress a protected metric beyond its tolerance.

Use `writing-skills` for skill-specific authoring and `release-engineering` for
production promotion. Never claim a score that was not read from an actual
result artifact.

## Completion condition

Cases, environment, baseline, candidate, artifacts, graders, costs, and limits
are reproducible; the decision follows predefined thresholds rather than a
post-hoc interpretation of the preferred result.
Files in this package (3)
  • SKILL.md 2 KB
  • agents/openai.yaml 1 KB
  • references/eval-schema.md 2 KB

Versions

RepositoryScoreLicence
This page · thiientv/godmode90MIT
FrancoStino/opencode-skills-collection100MIT
helx0071/skillsmo100MIT
alebgl77/claude-inc100MIT
guia-matthieu/clawfu-skills100MIT
seb1n/awesome-ai-agent-skills100MIT
charlieviettq/awesome-agent-skill100MIT
viktorbezdek/skillstack100MIT
mlflow/skills95Apache-2.0

Database snapshot: 13.11.2026 · package version 7e68c00aa132 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100evals
    Build and run evaluations for LLM features, prompts, agents, and RAG pipelines — golden datasets, assertions, LLM-as-judge graders, regression gates in CI, A/B comparison of prompts or models. Use whenever someone changes a prompt, swaps a model, ships an AI feature, asks \"is…
  • 100evals
    Designing and maintaining evaluations for LLM prompts, agents, and Agent Skills: success criteria, test-set composition, code/LLM-judge/human graders, trajectory and tool-use grading, trigger evals, regression suites, eval saturation. WHEN: \"eval\", \"evaluation suite\",…
  • 90skill-evaluation-workbench
    Use when designing, running, debugging, or hardening deterministic eval suites for agent skills, prompts, tool workflows, or MCP-backed cases.
  • 100promptfoo-evaluation
    Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`,…
  • 100promptfoo-evaluation
    Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost…
  • 100langgraph-testing-evaluation
    Use this skill when you need to test or evaluate LangGraph/LangChain agents: writing unit or integration tests, generating test scaffolds, mocking LLM/tool behavior, running trajectory evaluation (match or LLM-as-judge), running LangSmith dataset evaluations, and comparing two…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.