Skip to content
Skills Library
AI & agents · Testing

agentic-evaluation-framework

/agentic-evaluation-framework

This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".

Licence
MIT
Last repository change
23.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
5
Scripts
2
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 23.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 2 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/pairwise_ranking.py, scripts/rubric_scorer.py

What it does and when to use it

This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".

Triggers from the description

evaluate LLM output qualityset up LLM-as-judgebuild an eval rubriccompare model outputs pairwisemeasure agent quality
Original author description

This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".

How it works

  1. Overview
  2. Clarify First
  3. Quick Start
  4. Tools Overview
  5. Workflows
  6. Scope & Limitations
  7. Reference Documentation
  8. Common Patterns
Original SKILL.md headings · 98 lines, 1,319 words
SKILL.md preview
---
name: agentic-evaluation-framework
description: >
  This skill should be used when the user asks to "evaluate LLM output quality",
  "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise",
  or "measure agent quality".
license: MIT + Commons Clause
metadata:
  version: 1.0.0
  author: borghei
  category: engineering
  domain: ai-engineering
  updated: 2026-06-29
  tags: [evaluation, llm-as-judge, rubrics, agents, quality]
---

# Agentic Evaluation Framework

> **Category:** Engineering
> **Domain:** AI Engineering

## Overview

Design and run trustworthy evaluations for LLM and agent outputs: pick the right grading method (programmatic check, LLM-as-judge, or human review), write a scoring rubric that judges can apply consistently, rank competing variants by pairwise comparison, and watch for the biases that quietly corrupt judge scores — position bias, verbosity bias, and self-preference. The goal is an eval that you can *trust enough to ship on*: calibrated against human labels, cheap enough to run on every change, and tracked alongside cost and latency so you never trade quality away by accident. This skill is model- and vendor-agnostic: it reasons about the evaluation *method*, not any one provider's API, and its scripts aggregate scores you have already collected — they never call a model.

## Clarify First

Before designing or running an evaluation, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used.

- **What "good" means:** the dimensions you care about (accuracy, helpfulness, safety, format, tool-use) and their relative weight (defines the rubric `criteria` and `weights`).
- **Grading method:** can a deterministic check decide it, do you need an LLM judge, or must a human review it? (selects programmatic vs `rubric_scorer.py` absolute scoring vs `pairwise_ranking.py` comparison vs human-in-the-loop).
- **Ground truth & budget:** do you have human-labeled examples to calibrate the judge against, and what cost/latency per eval run is acceptable? (sets calibration plan and the quality/cost/latency budget).

Ask about the two or three inputs that matter most for this request. If the user says "just draft it," proceed and list your assumptions.

## Quick Start

```bash
cd engineering/agentic-evaluation-framework

# 1. Score outputs against a weighted rubric + check inter-rater agreement
python scripts/rubric_scorer.py --data rubric_scores.json

# 2. Rank competing variants from pairwise (A-vs-B) judgements
python scripts/pairwise_ranking.py --data pairwise_matches.json

# JSON output for piping into a dashboard or CI gate
python scripts/rubric_scorer.py --data rubric_scores.json --json
```

## Tools Overview

| Tool | Purpose | Key Flags |
|------|---------|-----------|
| `scripts/rubric_scorer.py` | Aggregate per-criterion scores into weighted totals, per-criterion means, pass/fail vs thresholds, and an inter-rater agreement metric | `--data`, `--json` |
| `scripts/pairwise_ranking.py` | Turn head-to-head win/loss records into a ranking via Elo + Bradley-Terry, plus a win-rate matrix | `--data`, `--k`, `--base`, `--json` |

Both scripts: Python 3 standard library only, argparse CLI, `--json` and human-readable output. They compute over scores you provide and never call a model. Run `--help` for full usage.

## Workflows

### 1. Build and calibrate an absolute-scoring rubric

1. Translate "what good means" into 3-6 named criteria, each with a weight, a 1-5 (or 1-7) scale, and a written anchor for every scale point (see `references/llm-judge-methodology.md`).
2. Have at least two graders — human, or human plus model — independently score a calibration set, and record the per-criterion scores as the `rubric_scorer.py` input JSON.
3. Run `rubric_scorer.py` and read `inter_rater_agreement`: low agreement means the rubric is ambiguous, not that a grader is wrong — tighten the anchors and re-score before trusting any number.
4. Once graders agree, treat the human scores as ground truth and check that the LLM judge's scores correlate; if not, revise the judge prompt or fall back to human review for that criterion.
5. Wire the passing rubric into CI as a gate (`--json` → pass/fail), and re-run agreement periodically to catch judge drift.

### 2. Rank model/prompt variants by pairwise comparison

1. When absolute scores are noisy, switch to pairwise: show the judge two outputs (A and B) for the same input and ask only "which is better?" — easier and more reliable than an absolute number.
2. Mitigate position bias by running each pair in both orders (A,B and B,A) and counting a win only if it survives both; record outcomes as `pairwise_ranking.py` matches, using `"winner": "tie"` for disagreements.
3. Run `pairwise_ranking.py` to get Elo and Bradley-Terry rankings plus the win-rate matrix; Bradley-Terry is order-independent and preferred for a fixed batch, Elo for a streaming sequence of matches.
4. Inspect the win-rate matrix for intransitivity (A>B, B>C, but C>A) — a sign of an unreliable judge or genuinely tied variants; collect more matches or add human adjudication.
5. Report the ranking next to cost and latency per variant so the "winner" is the best *quality-per-dollar-per-second*, not just the highest score.

## Scope & Limitations

**In Scope:**
- Choosing a grading method (programmatic check, LLM-as-judge or human review) for each quality dimension
- Weighted rubric scoring with `rubric_scorer.py`: weighted totals, per-criterion means, pass/fail against thresholds and an inter-rater agreement metric
- Pairwise ranking of variants with `pairwise_ranking.py` using Elo and Bradley-Terry, plus a win-rate matrix for spotting intransitive results
- Judge bias mitigation (position, verbosity, self-preference, sycophancy) and judge calibration against human labels, from the methodology and pitfalls references

**Out of Scope:**
- Calling a judge model or the model under test; scores and match outcomes are collected elsewhere and passed in as JSON
- Scenario suites with structural tool-call assertions, transcript replay and regression diffs (see `engineering/agent-harness/`)
- Writing or tuning the prompts being evaluated (see `engineering/prompt-engineer-toolkit/`)
- Measuring and cutting token spend (see `engineering/llm-cost-optimizer/`)

**Important Caveats:**
- The scripts aggregate numbers the user supplies. They cannot tell whether the underlying judgements were biased, careless or made on the training set.
- An LLM judge is only trustworthy after its scores agree with human labels on a held-out set. Agreement can drift when the judge model or prompt changes.
- Elo depends on match order and the K-factor. With few matches per pair, rankings from either method are unstable.
- The skill is vendor-agnostic and names no model. Judge model choice and pricing are left to the user.

## Reference Documentation

- **[references/llm-judge-methodology.md](references/llm-judge-methodology.md)** — rubric design and scale anchoring; absolute vs pairwise scoring; the judge-bias catalog (position, verbosity, self-preference, sycophancy) with concrete mitigations; calibrating a judge against human labels; the eval feedback loop (collect → grade → analyze → fix → regression-gate); and the quality/cost/latency metrics to track together.
- **[references/eval-pitfalls.md](references/eval-pitfalls.md)** — the anti-patterns that make evals lie: single-grader rubrics, judging on the training set, gameable metrics, ignoring variance, optimizing the judge instead of the model, and the decision table for when to use a programmatic check vs an LLM judge vs human review.

## Common Patterns

- **Cheapest valid grader wins** — if a deterministic check (regex, JSON-schema, unit test, exact match) can decide it, use that; reach for an LLM judge only for fuzzy quality, and human review only for high-stakes or judge-calibration work.
- **Pairwise over absolute when scores are noisy** — "which is better, A or B?" is more reliable than "rate this 1-5"; use absolute rubrics for thresholds/gates and pairwise for model selection.
- **Swap positions to kill position bias** — always run each comparison in both orders and only count wins that survive both; a variant that only wins in position A is a judge artifact.
- **Length is not quality** — strip or normalize for verbosity bias; a longer answer is not a better one, and judges systematically over-reward length unless you control for it.
- **Don't let a model grade its own homework** — self-preference bias means a model favors its own outputs; use a different judge family from the model under test, or anchor on human labels.
- **Calibrate before you trust** — a judge is only as good as its agreement with humans on a held-out set; measure that agreement first, then automate.
- **Track quality, cost, and latency as one number** — a quality win that triples cost or latency may be a net loss; always report the three together so the tradeoff is explicit.
- **Evals are regression tests for prompts** — freeze a labeled eval set, gate every prompt/model change on it, and grow the set from production failures you find.
Files in this package (5)
  • SKILL.md 9 KB
  • references/eval-pitfalls.md 5 KB
  • references/llm-judge-methodology.md 8 KB
  • scripts/pairwise_ranking.py 7 KB
  • scripts/rubric_scorer.py 7 KB

Versions

The database has no other skills named agentic-evaluation-framework.

Database snapshot: 13.11.2026 · package version ef7c782e18b4 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100agent-evaluation
    This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,…
  • 100evaluation
    This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", or mentions LLM-as-judge, multi-dimensional evaluation, agent testing, or quality gates for agent pipelines.
  • 100advanced-evaluation
    This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
  • 100advanced-evaluation
    This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
  • 100llm-as-judge
    Use an LLM as an evaluator for open-ended outputs — rubrics, pairwise comparison, calibration with human labels, bias mitigation. Covers when LLM-judge works, when it fails, and how to trust its scores. Use this skill when evaluating generative outputs at scale, building eval…
  • 90agent-evaluation
    为 Agent 建立或改进评估体系时使用——设计评估指标(Pass@k/Pass^k/成本)、搭建可重复运行的评估环境、选择确定性验证器或 LLM-as-a-Judge 及各自适用边界、编写 Rubric、做失败归因与回归任务、评估驱动的模型选型、判断统计显著性、建设可观测性与内部评估基础设施(消融、AB 测试、特性开关、提示词回归)、从 Benchmark 报告走到系统改进。

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.