eval-check
/eval-check
Evaluate the full harness configuration as a system. Scans all skills, commands, CLAUDE.md, and hooks for redundancy, overlap, type misclassification, and structural issues. Produces an informational report with restructuring suggestions. Use when the user wants to check their overall setup health, find redundant skills, detect overlapping triggers, or get restructuring recommendations before diving into individual skill evaluation. Triggers on "check my setup", "harness health", "are my skills redundant", "what should I merge", "setup overview", "configuration check".
- Author
- fullsend-ai ↗
- Repository
- agent-eval-harness ↗
- Licence
- Apache-2.0
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 4
- Scripts
- 2
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
- 2 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/harness_inventory.py, scripts/reference_checker.py
What it does and when to use it
Evaluate the full harness configuration as a system. Scans all skills, commands, CLAUDE.md, and hooks for redundancy, overlap, type misclassification, and structural issues. Produces an informational report with restructuring suggestions. Use when the user wants to check their overall setup health, find redundant skills, detect overlapping triggers, or get restructuring recommendations before diving into individual skill evaluation. Triggers on "check my setup", "harness health", "are my skills redundant", "what should I merge", "setup overview", "configuration check".
Triggers from the description
Original author description
Evaluate the full harness configuration as a system. Scans all skills, commands, CLAUDE.md, and hooks for redundancy, overlap, type misclassification, and structural issues. Produces an informational report with restructuring suggestions. Use when the user wants to check their overall setup health, find redundant skills, detect overlapping triggers, or get restructuring recommendations before diving into individual skill evaluation. Triggers on "check my setup", "harness health", "are my skills redundant", "what should I merge", "setup overview", "configuration check".
How it works
- Step 0: Parse Arguments
- Step 1: Inventory
- Step 2: Read All Skills
- Step 3: Read CLAUDE.md Files
- Step 4: Cross-Component Analysis
- Step 5: Generate Report
- Step 6: Present Summary
- Rules
SKILL.md preview
---
name: eval-check
description: Evaluate the full harness configuration as a system. Scans all skills, commands, CLAUDE.md, and hooks for redundancy, overlap, type misclassification, and structural issues. Produces an informational report with restructuring suggestions. Use when the user wants to check their overall setup health, find redundant skills, detect overlapping triggers, or get restructuring recommendations before diving into individual skill evaluation. Triggers on "check my setup", "harness health", "are my skills redundant", "what should I merge", "setup overview", "configuration check".
user-invocable: true
allowed-tools: Read, Bash, Glob, Grep, Agent, AskUserQuestion, Write
---
You are a harness health checker. You scan the full configuration (skills, commands, CLAUDE.md, hooks) as a single system and produce an informational report with findings and suggestions. You do not modify any files. You do not evaluate individual skill execution quality (that is what `/eval-run` does). Your focus is on the relationships between components: redundancy, overlap, type misclassification, and structural issues.
All findings are informational suggestions. The user decides what to act on.
## Step 0: Parse Arguments
| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `--output <path>` | no | `harness-report.md` | Where to write the report |
| `--include-global` | no | false | Also scan `~/.claude/CLAUDE.md` (user-global config, may contain personal instructions) |
## Step 1: Inventory
Run the inventory script to discover all configuration artifacts:
```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/harness_inventory.py --root .
```
This reports:
- Number of skills, commands, hooks
- Number of eval configs (`eval.yaml` files naming a skill; generated Harbor task packages are excluded)
- Approximate word count per skill (note: this is a word count, not a precise token count)
- Structural warnings (missing CLAUDE.md, skills without frontmatter descriptions)
If only one skill is found, report the inventory and skip to Step 6. Note: "Single-skill configuration. Cross-component analysis is not applicable." A single skill has no peers to overlap with, so the remaining analysis steps would produce no findings.
## Step 2: Read All Skills
For each skill found in Step 1, read its full SKILL.md content. Extract:
- The YAML frontmatter (between `---` delimiters): name, description, allowed-tools
- The body content: what rules and instructions it contains
- Any references to other skills (Skill tool calls, `/skill-name` references)
Keep a structured summary of each skill's domain, trigger description, and key rules.
## Step 3: Read CLAUDE.md Files
Scan CLAUDE.md files that exist in the project:
- `./CLAUDE.md` (project-level)
- `./.claude/CLAUDE.md` (project config directory)
If `--include-global` was passed, also read `~/.claude/CLAUDE.md`. This file may contain personal preferences or credential references, so it is opt-in only. If not passed, note in the report: "User-global CLAUDE.md was not scanned. Use `--include-global` to include it."
For each CLAUDE.md found, extract the rules and instructions it contains.
## Step 4: Cross-Component Analysis
Compare all skills against each other and against CLAUDE.md. Produce findings in these categories:
### 4a. Content Overlap
For each pair of skills, compare their content domains. Flag pairs where:
- Both skills cover the same programming language or domain (e.g., both cover Python error handling)
- Both skills contain similar or identical rules
- Both skills reference the same files or patterns
For each overlap found, note which rules are duplicated and the approximate word cost of the duplication.
### 4b. Trigger Overlap
Compare the frontmatter `description` fields (which control when skills activate). Flag pairs where:
- Both descriptions would activate for the same user task
- One description is a subset of the other (e.g., "Python development" vs "Python API client patterns")
- Both use broad, generic trigger language that would cause simultaneous loading
### 4c. CLAUDE.md Duplication
Compare each skill's rules against the CLAUDE.md content. Flag cases where:
- A skill's rule is already stated in CLAUDE.md (the rule loads every session via CLAUDE.md regardless of whether the skill activates)
- A skill contradicts a CLAUDE.md rule
### 4d. Type Misclassification
Evaluate whether each component is assigned to the correct mechanism:
- A skill whose rules must execute every time without exception should be in CLAUDE.md or a hook
- A skill that describes a specific user-triggered workflow should be a command
- A CLAUDE.md section with domain-specific rules that apply only sometimes should be a skill
- A skill with a deterministic check that should block an action should be a hook
### 4e. Structural Issues
Flag any issues found by the inventory script, plus:
- Skills without `description` in frontmatter (hurts trigger precision)
- Skills with very broad trigger descriptions that overlap with many peers
- Custom commands that shadow built-in Claude Code commands
### 4f. Reference Validation
Run the reference checker to find broken cross-component references:
```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/reference_checker.py --root . --format yaml
```
The script exits 0 when all references resolve, or 1 if any are broken or any script is missing. Unresolved and orphan findings are informational and never affect the exit code. This checks:
- **Cross-component references**: an explicit `skills/name/` path, or the phrase "Skill tool to invoke /name", must resolve to a known skill or command — otherwise it is **broken**. A bare backticked `` `/name` `` is lower confidence, since it is indistinguishable from a Claude Code built-in (the `help` or `model` command), another plugin's command, or a filesystem path; an unresolved one is listed under **unresolved references** instead. Matches inside a URL are ignored — a link to someone else's repo often contains a `skills/<name>/` segment.
- **Script references**: when a SKILL.md references `${CLAUDE_SKILL_DIR}/scripts/foo.py` or a cross-skill path via `${CLAUDE_PLUGIN_ROOT}`, verify the script exists. A path that resolves outside the project (shared plugin dir, monorepo sibling) cannot be checked from here, so it is listed under **unresolved references** — surfaced, but not treated as missing.
- **Eval config references**: when an eval.yaml with `runner.type: claude-code` names an `execution.skill`, verify that skill exists. Non-claude-code runners are skipped (there `skill` is only a label for the command template), as are eval.yaml copies bundled inside generated Harbor task packages. Plugin-namespaced names (`plugin.skill`, `plugin:skill`) resolve against either the full name or the local segment.
- **Orphan skills**: skills not referenced by any other component (only flagged when >3 skills exist; skills with `user-invocable: true` in frontmatter are exempt)
## Step 5: Generate Report
Write the report to the path specified by `--output` (default: `harness-report.md`) only if it resolves within the project root. If it resolves outside root (e.g., `..` traversal or absolute external path), refuse and ask for a valid path. Use the Write tool.
Structure the report as:
```markdown
# Harness Health Report
Generated: <date>
## Inventory
- Skills: N (total ~X words)
- Commands: N
- Hooks: N
- Eval configs: N
- CLAUDE.md: Yes/No
### Skills by size
| Skill | Words | Description |
|-------|-------|-------------|
| ... | ... | ... |
## Findings
### Content Overlap
(list findings, or "No content overlap detected.")
### Trigger Overlap
(list findings, or "No trigger overlap detected.")
### CLAUDE.md Duplication
(list findings, or "No CLAUDE.md duplication detected.")
### Type Misclassification
(list findings, or "No type misclassification detected.")
### Structural Issues
(list findings, or "No structural issues detected.")
### Reference Validation
(list broken references and missing scripts, or "All references resolve." Then list unresolved
references and orphan skills separately, noting they are informational — an unresolved `/name` is
often a Claude Code built-in or a command from another plugin.)
## Suggestions
Numbered list of concrete, actionable suggestions. Each suggestion:
1. What to do (merge, move, rename, narrow, remove)
2. Which components are involved
3. Why (which finding it addresses)
## Notes
- Word counts are approximate (whitespace-split). Actual token counts will differ.
- Overlap detection is based on content comparison by the reviewing model. Results are informational, not deterministic.
- User-global CLAUDE.md was [scanned / not scanned].
```
## Step 6: Present Summary
Show the user a brief terminal summary:
- Total components found
- Number of findings per category
- Top 3 most actionable suggestions
- Where the full report was saved
Suggest next steps:
- `/eval-analyze --skill <name>` to dive deeper into a specific skill
- `/eval-run --model <model>` to test whether a skill produces measurably different output
- Review and act on the suggestions in the report
## Rules
- **Read-only.** Do not modify any skill, command, CLAUDE.md, or hook file. Write only the report.
- **Informational, not prescriptive.** All findings are suggestions. The user decides what to act on.
- **Skip unreadable files.** If a file can't be read, note it in the report and continue. Don't fail the whole report for one missing file.
- **No false precision.** Word counts are approximate. Overlap assessment is qualitative. Don't present LLM judgments as deterministic measurements.
- **Respect privacy.** Only scan `~/.claude/CLAUDE.md` if `--include-global` is explicitly passed.
$ARGUMENTS
Files in this package (4)
- SKILL.md 10 KB
- scripts/agent_eval 1 KB
- scripts/harness_inventory.py 10 KB
- scripts/reference_checker.py 19 KB
Versions
The database has no other skills named eval-check.
Database snapshot: 13.11.2026 · package version aebacb2f1b42 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100harness
Stage map and operating rules for building or improving a Claude Code harness (skills, agents, commands, hooks, CLAUDE.md, scripts) with research, evals against a no-harness baseline, and cost optimization. Use at the start of any harness-building session, when the user asks to… - 100harness-design
Decide where each piece of a Claude Code harness belongs (CLAUDE.md, rules, skills, hooks, subagents, scripts, MCP) with token budgets, then build the files from templates. Use when designing or building skills, agents, commands, hooks or CLAUDE.md for a harness or plugin, when… - 100setup-project
Analyze the current repository and propose a lean Claude Code setup (CLAUDE.md, verification commands, optional hooks, skills, subagents). Asks for approval before changing anything. - 100cmd-skills-review
Audit personal skills for redundancy, verbosity, weak triggers, and overlap. Runs a Claude→Codex review loop, presents per-item approval checkboxes, then applies approved edits and updates README and agent metadata. Use when asked to "review my skills", "audit my skills",… - 100harness-eval
Evaluate a repo agent harness (AGENTS.md, rules, skills, skill refs) for broken paths/commands, redundant instructions, and usefulness using a stack-agnostic dual-judge protocol with planted traps. HIGH PRIORITY questionnaires at top: Q1 optional docs, Q2 B/C budget before Track… - 100skills-eval
Evaluate Claude skill quality through auditing. Use when reviewing or auditing skills.