agentic-evaluation-framework
/agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
- Author
- borghei ↗
- Repository
- Claude-Skills ↗
- Licence
- MIT
- Last repository change
- 23.10.2026
Assessment and security
- Files in this package
- 5
- Scripts
- 2
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 23.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 2 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/pairwise_ranking.py, scripts/rubric_scorer.py
What it does and when to use it
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
Triggers from the description
Original author description
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
How it works
- Overview
- Clarify First
- Quick Start
- Tools Overview
- Workflows
- Scope & Limitations
- Reference Documentation
- Common Patterns
SKILL.md preview
--- name: agentic-evaluation-framework description: > This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality". license: MIT + Commons Clause metadata: version: 1.0.0 author: borghei category: engineering domain: ai-engineering updated: 2026-06-29 tags: [evaluation, llm-as-judge, rubrics, agents, quality] --- # Agentic Evaluation Framework > **Category:** Engineering > **Domain:** AI Engineering ## Overview Design and run trustworthy evaluations for LLM and agent outputs: pick the right grading method (programmatic check, LLM-as-judge, or human review), write a scoring rubric that judges can apply consistently, rank competing variants by pairwise comparison, and watch for the biases that quietly corrupt judge scores — position bias, verbosity bias, and self-preference. The goal is an eval that you can *trust enough to ship on*: calibrated against human labels, cheap enough to run on every change, and tracked alongside cost and latency so you never trade quality away by accident. This skill is model- and vendor-agnostic: it reasons about the evaluation *method*, not any one provider's API, and its scripts aggregate scores you have already collected — they never call a model. ## Clarify First Before designing or running an evaluation, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used. - **What "good" means:** the dimensions you care about (accuracy, helpfulness, safety, format, tool-use) and their relative weight (defines the rubric `criteria` and `weights`). - **Grading method:** can a deterministic check decide it, do you need an LLM judge, or must a human review it? (selects programmatic vs `rubric_scorer.py` absolute scoring vs `pairwise_ranking.py` comparison vs human-in-the-loop). - **Ground truth & budget:** do you have human-labeled examples to calibrate the judge against, and what cost/latency per eval run is acceptable? (sets calibration plan and the quality/cost/latency budget). Ask about the two or three inputs that matter most for this request. If the user says "just draft it," proceed and list your assumptions. ## Quick Start ```bash cd engineering/agentic-evaluation-framework # 1. Score outputs against a weighted rubric + check inter-rater agreement python scripts/rubric_scorer.py --data rubric_scores.json # 2. Rank competing variants from pairwise (A-vs-B) judgements python scripts/pairwise_ranking.py --data pairwise_matches.json # JSON output for piping into a dashboard or CI gate python scripts/rubric_scorer.py --data rubric_scores.json --json ``` ## Tools Overview | Tool | Purpose | Key Flags | |------|---------|-----------| | `scripts/rubric_scorer.py` | Aggregate per-criterion scores into weighted totals, per-criterion means, pass/fail vs thresholds, and an inter-rater agreement metric | `--data`, `--json` | | `scripts/pairwise_ranking.py` | Turn head-to-head win/loss records into a ranking via Elo + Bradley-Terry, plus a win-rate matrix | `--data`, `--k`, `--base`, `--json` | Both scripts: Python 3 standard library only, argparse CLI, `--json` and human-readable output. They compute over scores you provide and never call a model. Run `--help` for full usage. ## Workflows ### 1. Build and calibrate an absolute-scoring rubric 1. Translate "what good means" into 3-6 named criteria, each with a weight, a 1-5 (or 1-7) scale, and a written anchor for every scale point (see `references/llm-judge-methodology.md`). 2. Have at least two graders — human, or human plus model — independently score a calibration set, and record the per-criterion scores as the `rubric_scorer.py` input JSON. 3. Run `rubric_scorer.py` and read `inter_rater_agreement`: low agreement means the rubric is ambiguous, not that a grader is wrong — tighten the anchors and re-score before trusting any number. 4. Once graders agree, treat the human scores as ground truth and check that the LLM judge's scores correlate; if not, revise the judge prompt or fall back to human review for that criterion. 5. Wire the passing rubric into CI as a gate (`--json` → pass/fail), and re-run agreement periodically to catch judge drift. ### 2. Rank model/prompt variants by pairwise comparison 1. When absolute scores are noisy, switch to pairwise: show the judge two outputs (A and B) for the same input and ask only "which is better?" — easier and more reliable than an absolute number. 2. Mitigate position bias by running each pair in both orders (A,B and B,A) and counting a win only if it survives both; record outcomes as `pairwise_ranking.py` matches, using `"winner": "tie"` for disagreements. 3. Run `pairwise_ranking.py` to get Elo and Bradley-Terry rankings plus the win-rate matrix; Bradley-Terry is order-independent and preferred for a fixed batch, Elo for a streaming sequence of matches. 4. Inspect the win-rate matrix for intransitivity (A>B, B>C, but C>A) — a sign of an unreliable judge or genuinely tied variants; collect more matches or add human adjudication. 5. Report the ranking next to cost and latency per variant so the "winner" is the best *quality-per-dollar-per-second*, not just the highest score. ## Scope & Limitations **In Scope:** - Choosing a grading method (programmatic check, LLM-as-judge or human review) for each quality dimension - Weighted rubric scoring with `rubric_scorer.py`: weighted totals, per-criterion means, pass/fail against thresholds and an inter-rater agreement metric - Pairwise ranking of variants with `pairwise_ranking.py` using Elo and Bradley-Terry, plus a win-rate matrix for spotting intransitive results - Judge bias mitigation (position, verbosity, self-preference, sycophancy) and judge calibration against human labels, from the methodology and pitfalls references **Out of Scope:** - Calling a judge model or the model under test; scores and match outcomes are collected elsewhere and passed in as JSON - Scenario suites with structural tool-call assertions, transcript replay and regression diffs (see `engineering/agent-harness/`) - Writing or tuning the prompts being evaluated (see `engineering/prompt-engineer-toolkit/`) - Measuring and cutting token spend (see `engineering/llm-cost-optimizer/`) **Important Caveats:** - The scripts aggregate numbers the user supplies. They cannot tell whether the underlying judgements were biased, careless or made on the training set. - An LLM judge is only trustworthy after its scores agree with human labels on a held-out set. Agreement can drift when the judge model or prompt changes. - Elo depends on match order and the K-factor. With few matches per pair, rankings from either method are unstable. - The skill is vendor-agnostic and names no model. Judge model choice and pricing are left to the user. ## Reference Documentation - **[references/llm-judge-methodology.md](references/llm-judge-methodology.md)** — rubric design and scale anchoring; absolute vs pairwise scoring; the judge-bias catalog (position, verbosity, self-preference, sycophancy) with concrete mitigations; calibrating a judge against human labels; the eval feedback loop (collect → grade → analyze → fix → regression-gate); and the quality/cost/latency metrics to track together. - **[references/eval-pitfalls.md](references/eval-pitfalls.md)** — the anti-patterns that make evals lie: single-grader rubrics, judging on the training set, gameable metrics, ignoring variance, optimizing the judge instead of the model, and the decision table for when to use a programmatic check vs an LLM judge vs human review. ## Common Patterns - **Cheapest valid grader wins** — if a deterministic check (regex, JSON-schema, unit test, exact match) can decide it, use that; reach for an LLM judge only for fuzzy quality, and human review only for high-stakes or judge-calibration work. - **Pairwise over absolute when scores are noisy** — "which is better, A or B?" is more reliable than "rate this 1-5"; use absolute rubrics for thresholds/gates and pairwise for model selection. - **Swap positions to kill position bias** — always run each comparison in both orders and only count wins that survive both; a variant that only wins in position A is a judge artifact. - **Length is not quality** — strip or normalize for verbosity bias; a longer answer is not a better one, and judges systematically over-reward length unless you control for it. - **Don't let a model grade its own homework** — self-preference bias means a model favors its own outputs; use a different judge family from the model under test, or anchor on human labels. - **Calibrate before you trust** — a judge is only as good as its agreement with humans on a held-out set; measure that agreement first, then automate. - **Track quality, cost, and latency as one number** — a quality win that triples cost or latency may be a net loss; always report the three together so the tradeoff is explicit. - **Evals are regression tests for prompts** — freeze a labeled eval set, gate every prompt/model change on it, and grow the set from production failures you find.
Files in this package (5)
- SKILL.md 9 KB
- references/eval-pitfalls.md 5 KB
- references/llm-judge-methodology.md 8 KB
- scripts/pairwise_ranking.py 7 KB
- scripts/rubric_scorer.py 7 KB
Versions
The database has no other skills named agentic-evaluation-framework.
Database snapshot: 13.11.2026 · package version ef7c782e18b4 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100agent-evaluation
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,… - 100evaluation
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", or mentions LLM-as-judge, multi-dimensional evaluation, agent testing, or quality gates for agent pipelines. - 100advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment. - 100advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment. - 100llm-as-judge
Use an LLM as an evaluator for open-ended outputs — rubrics, pairwise comparison, calibration with human labels, bias mitigation. Covers when LLM-judge works, when it fails, and how to trust its scores. Use this skill when evaluating generative outputs at scale, building eval… - 90agent-evaluation
为 Agent 建立或改进评估体系时使用——设计评估指标(Pass@k/Pass^k/成本)、搭建可重复运行的评估环境、选择确定性验证器或 LLM-as-a-Judge 及各自适用边界、编写 Rubric、做失败归因与回归任务、评估驱动的模型选型、判断统计显著性、建设可观测性与内部评估基础设施(消融、AB 测试、特性开关、提示词回归)、从 Benchmark 报告走到系统改进。