Skip to content
Skills Library
AI & agents · Testing

evaluate-language-model

/evaluate-language-model

Design repeatable language-model capability, quality, behavior, regression, or safety evaluations. Use when creating cases and graders, comparing a treatment with a baseline, or deciding whether a checkpoint passes.

Repository
socket ↗
Licence
Apache-2.0
Last repository change
26.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
6
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 26.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Design repeatable language-model capability, quality, behavior, regression, or safety evaluations. Use when creating cases and graders, comparing a treatment with a baseline, or deciding whether a checkpoint passes.

Original author description

Design repeatable language-model capability, quality, behavior, regression, or safety evaluations. Use when creating cases and graders, comparing a treatment with a baseline, or deciding whether a checkpoint passes.

How it works

  1. Define The Claim
  2. Build The Evaluation
  3. Choose The Harness
  4. Evidence Classes
  5. Resources
Original SKILL.md headings · 39 lines, 304 words
SKILL.md preview
---
name: evaluate-language-model
description: Design repeatable language-model capability, quality, behavior, regression, or safety evaluations. Use when creating cases and graders, comparing a treatment with a baseline, or deciding whether a checkpoint passes.
---

# Evaluate Language Model

## Define The Claim

State the population, task, model artifact, prompt/template, decoding settings, and decision threshold. A benchmark score supports only the conditions it measured.

## Build The Evaluation

1. Separate development cases from a held-out decision set.
2. Give every case a stable ID, inputs, tags, expected behavior, and scoring method. Start from `assets/eval-cases.jsonl`.
3. Prefer executable or deterministic graders for exact structure, tests, schemas, and calculations.
4. For model judges, pin the judge model/revision, rubric, prompt, sampling settings, and parser. Calibrate against human-labeled examples and test order/position effects.
5. Record refusals, invalid outputs, timeouts, and grader failures as outcomes; do not silently drop them.
6. Run baseline and treatment on the same cases and settings.
7. Preserve per-case results, require identical case IDs for decision comparisons, then aggregate by meaningful slices as well as globally. Use partial comparisons only as explicitly labeled diagnostics.
8. Quantify uncertainty with repeated runs, confidence intervals, or paired tests appropriate to the metric.
9. Inspect regressions and disagreements before accepting the aggregate result.

## Choose The Harness

- Use [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) for supported standardized, mostly model-in/model-out academic benchmarks such as MMLU; pin the task and harness revisions.
- Use [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) when the evaluation needs sandboxed code/tools, multi-step state, agent behavior, explicit scorers, or stronger execution isolation.
- Use a small project-native runner when neither framework adds useful capability. Preserve the same case, grader, provenance, and raw-result contract.

## Evidence Classes

- Configuration validation: cases and graders parse.
- Smoke evidence: a small subset completes end to end.
- Evaluation evidence: the frozen set completes with retained per-case outputs.
- Decision evidence: thresholds, uncertainty, regressions, and limitations support the stated choice.

## Resources

- `assets/eval-cases.jsonl`: starter case schema.
- `assets/evaluation-report.md`: comparison report template.
- `references/evaluation-methods.md`: grader and uncertainty rules.
- `scripts/compare-eval-runs.fsx`: paired JSONL comparison.
Files in this package (6)
  • SKILL.md 3 KB
  • agents/openai.yaml 1 KB
  • assets/eval-cases.jsonl 1 KB
  • assets/evaluation-report.md 1 KB
  • references/evaluation-methods.md 1 KB
  • scripts/compare-eval-runs.fsx 4 KB

Versions

The database has no other skills named evaluate-language-model.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version a269f6987ede (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100agent-evaluation
    Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Use when defining agent quality, comparing prompts or models, validating a release, measuring tool-use…
  • 100harness-eval
    Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or…
  • 100anthropic-evaluations
    This skill should be used when the user asks to "create evals", "evaluate an agent", "build evaluation suite", or mentions agent testing, graders, or benchmarks. Also suggest when building coding agents, conversational agents, or research agents that need quality assurance.
  • 100prompt-engineer
    Use this agent when you need to design, optimize, test, or evaluate prompts for large language models in production systems.
  • 100prompt-engineer
    Use this agent when you need to design, optimize, test, or evaluate prompts for large language models in production systems.
  • 84create-eval-set
    Create a test set CSV file for import into Copilot Studio's in-product Evaluate tab. Reads the agent's topics, instructions, and knowledge sources to generate meaningful test cases with appropriate graders (General quality, Compare meaning, Exact match, etc.). Use when the user…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.