eval-analysis
/eval-analysis
Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment.
- Author
- eforge-build ↗
- Repository
- eforge ↗
- Licence
- Apache-2.0
- Last repository change
- 13.07.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 13.07.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment.
Triggers from the description
Original author description
Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment.
How it works
- Prerequisites
- Workflow
- Important Reminders
SKILL.md preview
--- name: eval-analysis description: Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment. --- # /eval-analysis Structured methodology for analyzing eforge eval results, identifying signal patterns, and proposing changes with anti-bias safeguards. ## Prerequisites - The eval harness MCP server must be connected (provides `eval_runs`, `eval_observations`, `eval_scenario_detail`, `eval_run`, `eval_results` tools) - You should be in or have access to the eforge project codebase ## Workflow ### Step 1: Gather Recent Eval Data Start by checking what eval data is available: 1. Use `eval_runs` to list recent eval runs. Note run IDs, timestamps, and which scenarios were included. 2. If the user mentions a specific run or comparison, use `eval_results` with the relevant run ID(s). 3. If comparing baseline vs candidate, use `eval_results` with the `compare` parameter to get a structured regression comparison between two runs. ### Step 2: Pull Observations For runs of interest, use `eval_observations` to get detailed per-scenario observation data. This gives you the raw signal - scores, pass/fail, and any metadata attached to each observation. Focus on: - Scenarios with low scores or failures - Scenarios where scores changed significantly between runs - Patterns across scenario categories ### Step 3: Drill Into Affected Scenarios For each scenario showing issues or regressions, use `eval_scenario_detail` to get the full scenario specification - inputs, expected outputs, scoring criteria, and any notes. This is critical context: you need to understand what the scenario tests before reasoning about why it failed. ### Step 4: Read Relevant Source Before forming hypotheses about root cause, **always read the relevant eforge prompt or config file** that the failing scenarios exercise. Do not guess at prompt content from memory. Common locations: - Agent prompts: `src/engine/prompts/*.md` - Agent implementations: `src/engine/agents/*.ts` - Config and defaults: `src/engine/config.ts` - Schema definitions: `src/engine/schemas.ts` - Pipeline stages: `src/engine/pipeline.ts` Understanding the actual prompt/config text is essential for accurate root cause analysis. ### Step 5: Apply Anti-Bias Gating Rules Before proposing any change based on eval data, check every proposed action against all five gating rules. A proposal that violates any rule must be revised or dropped. #### Rule 1: Never Reduce Sensitivity Do not propose changes that would make the system less sensitive to errors, edge cases, or subtle signals. If a scenario is failing because the agent catches something it previously missed, that may be a feature, not a bug. Reducing sensitivity to make numbers go up is a false improvement. #### Rule 2: Never Shift Threshold in One Direction Do not propose adjusting a threshold, cutoff, or scoring boundary that only moves it in one direction (e.g., making pass criteria more lenient, or raising a tolerance). Any threshold change must be justified by evidence that the current threshold is wrong in both directions - too strict in some cases AND too lenient in others - or must be accompanied by a new scenario that tests the boundary from the other side. #### Rule 3: Always Require Counter-Scenario For any proposed behavior change, identify at least one existing scenario (or propose a new one) that would regress if the change were applied incorrectly. If you cannot identify a counter-scenario, the change is under-constrained and should not be made until one exists. The counter-scenario proves the change is targeted rather than a broad relaxation. #### Rule 4: Prefer Scenario Additions Over Behavior Changes When eval results reveal a gap, the first response should be "do we need a new scenario?" not "do we need to change the prompt?" Adding scenarios that capture the desired behavior is lower-risk than modifying prompts or config. Only propose behavior changes when the scenario coverage is already adequate and the root cause is clearly in the prompt/config. #### Rule 5: Respect Confidence Thresholds Do not propose changes based on small sample sizes or marginal differences. Require: - At least 3 observations of a pattern before treating it as signal (not noise) - A score difference of at least 10% between baseline and candidate to call it a regression or improvement - If a scenario has high variance across runs, flag it as unreliable rather than acting on its latest result ### Step 6: Formulate Findings Present findings conversationally with three components for each issue: 1. **Data** - What the eval numbers show. Quote specific scores, scenario names, and run IDs. Show the delta if comparing runs. 2. **Hypothesis** - What you think is causing the observed behavior, grounded in the source you read in Step 4. Be explicit about confidence level (high/medium/low) and what evidence supports or weakens the hypothesis. 3. **Proposed Action** - What to do about it. This must pass all five gating rules. Clearly label whether the action is: - **Add scenario** - New eval scenario to improve coverage - **Modify prompt/config** - Change to eforge source (with specific file and section) - **Investigate further** - Need more data before acting - **No action** - Signal is noise or expected behavior ### Step 7: Regression Gate (for candidate comparison) When comparing a candidate change against a baseline: 1. Use `eval_results` with the `compare` parameter to get the structured comparison. 2. Flag any scenario that regressed by more than 10% as a blocker. 3. For each regression, trace through Steps 3-5 to determine if the regression is: - **Real** - The candidate change broke something. Block the change. - **Flaky** - The scenario has high variance. Flag but don't block. - **Expected** - The candidate intentionally changes behavior the scenario tests. Update the scenario. 4. Summarize with a clear **PASS/BLOCK** recommendation. ### Step 8: Run New Evals (if needed) If the analysis suggests running a new eval (e.g., after adding scenarios or to get more data): 1. Use `eval_run` to kick off a new run. 2. Wait for completion, then loop back to Step 1 with the new run data. ## Important Reminders - Always read source before reasoning about root cause (Step 4) - Every proposed change must pass all five gating rules (Step 5) - Present findings as data + hypothesis + action (Step 6) - When in doubt, propose adding scenarios rather than changing behavior (Rule 4) - Small sample sizes are noise, not signal (Rule 5)
Versions
The database has no other skills named eval-analysis.
Database snapshot: 13.11.2026 · package version fa94a6ce479d (SHA-256 digest of the entire package)
Downloads in the last 30 days: 2 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100evals-analyze
Analyze LLM eval results — score breakdowns by category, regression detection vs baseline, failure clustering. Use when asked to "analyze our eval results", "did the model regress", or "cluster the eval failures". - 100skill-eval
Runs and analyzes eval test cases for Claude skills. Use when asked to "test a skill", "check if this skill works", "run evals", "verify skill quality", or after creating/modifying a skill. - 95eval-analyzer
Identify AILANG language gaps from agent struggles, analyze eval baselines, and generate actionable insights. PRIMARY PURPOSE is finding what stdlib/prompt improvements would help agents succeed. Use when analyzing eval results, checking benchmarks, or investigating failures. - 95eval-analyzer
Identify AILANG language gaps from agent struggles, analyze eval baselines, and generate actionable insights. PRIMARY PURPOSE is finding what stdlib/prompt improvements would help agents succeed. Use when analyzing eval results, checking benchmarks, or investigating failures. - 100evals
Author an Interlock eval case from an observed model-behaviour failure, and read a completed eval results run. Use when you have a reproduced failure — an archived proposal, a changelog entry, a run artifact, or a transcript — worth regression-testing against a real model, or… - 100output-eval-error-analysis
Systematically review workflow traces to identify failure modes before building evaluators. Use when starting an eval project, after significant pipeline changes, or when production quality drops.