Skip to content
Skills Library
AI & agents · Analysis

eval-error-analysis

/eval-error-analysis

Cluster human review annotations into evidence-linked failure modes, refine a taxonomy with the reviewer, and prioritize errors. Use when review notes already exist, especially for local workflows; this does not generate validated grader labels.

Repository
eval-skills ↗
Licence
Apache-2.0
Last repository change
16.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
17
Scripts
5
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 16.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What to check before installing
  • 5 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/setup_workspace.py, templates/review-app/app.js, templates/review-app/server.mjs, templates/review-app/server.py, templates/sample_traces.py

What it does and when to use it

Cluster human review annotations into evidence-linked failure modes, refine a taxonomy with the reviewer, and prioritize errors. Use when review notes already exist, especially for local workflows; this does not generate validated grader labels.

Original author description

Cluster human review annotations into evidence-linked failure modes, refine a taxonomy with the reviewer, and prioritize errors. Use when review notes already exist, especially for local workflows; this does not generate validated grader labels.

How it works

  1. Start from reviewed evidence
  2. Propose and refine
  3. Prioritize and hand over
Original SKILL.md headings · 30 lines, 421 words
SKILL.md preview
---
name: eval-error-analysis
description: Cluster human review annotations into evidence-linked failure modes, refine a taxonomy with the reviewer, and prioritize errors. Use when review notes already exist, especially for local workflows; this does not generate validated grader labels.
license: Apache-2.0
metadata:
  author: Confident AI
  version: "1.1.0"
---

# Turn annotations into failure modes

Read [the operating agreement](references/workflow.md) and [analysis procedure and artifacts](references/error-analysis.md). For local work, use [local review guidance](references/local-review.md); for an existing managed workflow, use [managed guidance](references/managed.md).

For bundled local tools and exact commands, read [toolkit setup](references/toolkit.md). Run [the workspace installer](scripts/setup_workspace.py) for this skill; it preserves existing customized files.
For the managed route, use [concrete service operations](references/api-managed.md) after confirming the connected schema.

## Start from reviewed evidence

Locate existing annotations, their source traces, sampling rules, and any taxonomy revisions. Preserve original notes and author/status information. Inspect the behavior behind the notes before grouping it. If there are no human observations, prepare a review queue and use `eval-discover` if installed; agent-generated observations remain suggestions.

Reuse a managed platform's existing analysis and exports when available. For either local track, perform the analysis over local records and extend the existing review UI. No grader framework or new hosted connection is required.

## Propose and refine

Group notes by the observable failure and expected behavior. Use semantic search or embeddings only to find candidate neighbors; inspect evidence to decide membership. Similar wording can describe different failures, and different wording can describe the same failure.

Give each proposed mode a stable ID, clear definition, inclusion/exclusion rules, representative evidence, and boundary examples. Allow multiple modes per annotation and an unresolved queue. Separate observed symptoms from hypothesized root causes. Do not force every record into a category or turn a broad label such as “hallucination” into a supposedly actionable taxonomy.

Present groups beside original annotations and traces. Let the reviewer merge, split, rename, move, reject, and confirm proposals. Preserve revision history and reviewer decisions. Human approval of a category does not automatically confirm every assignment or produce per-criterion Pass/Fail labels. If review is unavailable, finish the proposed taxonomy, evidence links, and review queue without claiming confirmation.

## Prioritize and hand over

Count distinct cases per mode and state the reviewed denominator and sampling purpose. Multi-label counts can overlap. Rank using observed frequency, product impact, and uncertainty; rare severe failures must remain visible. Do not infer production prevalence from a discovery sample.

Save a versioned taxonomy, annotation-to-mode mappings, unresolved items, and a concise analysis report using the reference's local artifact convention or an equivalent existing format. Verify evidence links and save/reload/export behavior when changing a UI. Report unverified interactions honestly.

Hand confirmed modes and examples to case curation and grader design (`eval-dataset` and `eval-grade` when installed). With a trusted baseline, use the prioritized development failures to guide `eval-descent`. Keep held-out evidence out of the optimization loop.
Files in this package (17)
  • LICENSE 11 KB
  • SKILL.md 4 KB
  • agents/openai.yaml 1 KB
  • references/api-managed.md 3 KB
  • references/error-analysis.md 6 KB
  • references/local-review.md 4 KB
  • references/managed.md 4 KB
  • references/portability.md 2 KB
  • references/toolkit.md 7 KB
  • references/workflow.md 5 KB
  • scripts/setup_workspace.py 2 KB
  • templates/sample_traces.py 6 KB
  • templates/review-app/app.css 6 KB
  • templates/review-app/app.js 20 KB
  • templates/review-app/index.html 3 KB
  • templates/review-app/server.mjs 10 KB
  • templates/review-app/server.py 14 KB

Versions

The database has no other skills named eval-error-analysis.

Database snapshot: 13.11.2026 · package version cc564d3b1982 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100output-eval-error-analysis
    Systematically review workflow traces to identify failure modes before building evaluators. Use when starting an eval project, after significant pipeline changes, or when production quality drops.
  • 100error-analysis
    Evals-first error analysis for LLM apps: clusters real Langfuse or JSONL traces into a human-confirmed failure taxonomy with counts, then recommends binary pass/fail evals for recurring named modes. Use to learn what to measure before writing evals. Not for CI failures.
  • 100trace-codes
    Turn free-form notes from reading agent traces into trace codes: named, yes/no failure or quality patterns with definitions, examples and counts. Use this skill whenever someone has annotated traces or review notes and asks to "cluster my notes", "create trace codes", "find the…
  • 100agent-error-taxonomy-root-cause-analysis
    Use this skill when classifying agent operational failures, tool exceptions, and hallucination patterns into a systematic error taxonomy.
  • 100orq-analyze-traces
    Analyze a live agent, deployment, or local agent from its production traces — relay its configuration and terminal states, then build a failure taxonomy by open coding and axial coding, and write it to an error-analysis file other skills read. Use when debugging agent or…
  • 100error-analysis
    Turn an AI feature''s traces or reviewers'' notes into counted, priced failure modes before any metric or judge. Use for unexplained bad answers, "what should we measure?", or notes to group.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.