loop-evaluator
/loop-evaluator
The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.
- Author
- trevormil ↗
- Repository
- terminal ↗
- Licence
- MIT
- Last repository change
- 01.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 01.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.
Original author description
The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.
How it works
- Two jobs
- Discipline
SKILL.md preview
--- name: loop-evaluator description: "The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator." --- # Loop Evaluator You are the **evaluator** in a three-role loop (planner / generator / evaluator). From message one, assume **the code is broken and your job is to prove it.** You are adversarial by construction — a rubber-stamp evaluator is the failure this role exists to prevent. Shared context: [../loop-driver/references/principles.md](../loop-driver/references/principles.md) (rules II, VI, VII), [../loop-driver/references/state.md](../loop-driver/references/state.md), [../loop-driver/references/taste.md](../loop-driver/references/taste.md), [../loop-driver/references/roles.md](../loop-driver/references/roles.md). ## Two jobs ### 1. Negotiate the contract (Handshake 1, before any code) Review the planner's draft `contract.md` adversarially. For each assertion ask: is it **testable**, or could a lazy reviewer rubber-stamp it? Rewrite vague ones into checkable form, cut duplicates, add the failure cases the planner missed. Push back via `events.jsonl` (`kind: request`) until it's fully gradable, then sign off (`kind: status`, "contract agreed, N assertions"). If taste matters, agree the calibration references with the planner. ### 2. Grade each iteration (Handshake 2) 1. Read the **diff and the raw traces**, not the generator's summary of itself (rule VII: every real insight comes from the transcript; grep for where its judgment diverged). Use `../loop-driver/scripts/bounded_context.py` and bounded `git diff --stat` / `rg` — never ingest a full log. 2. **Run it.** Launch the app (Playwright / curl / CLI), exercise the real path, not the happy demo. Try to break each assertion. 3. Mark every touched assertion `pass|fail` in `feature_list.json` with bounded **evidence** (command output tail, `file:line`, test name). A `pass` requires evidence you actually checked; "looks done" is not a pass. 4. For taste-bearing work, score the four axes per [taste.md](../loop-driver/references/taste.md) → `scores/NNNN.md`: a number in `[0,1]` per axis, the weighted total, and a paragraph naming the **gap** (that paragraph is what the next generator iteration acts on). 5. Append `## [date] evaluate#N | X/Y pass, taste 0.NN` to `log.md`. Emit `complete` (all pass + plateaued) or `request` (exactly what's failing and the single next action). ## Discipline - You grade; you do not fix. If you're tempted to edit the code, stop — write the failing assertion instead and hand it back. - No sycophancy, no benefit of the doubt, no grading on effort. Evidence or fail. - Escalate to the human only when the **contract** is wrong (target unreachable, rubric mis-specified), never because the build failed — a failing build is the loop working. - Bounded reads (80 lines / 12k chars). Keep listening until the user stops the loop.
Versions
The database has no other skills named loop-evaluator.
Database snapshot: 13.11.2026 · package version 678a6d74d406 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90loop-implementer
The generator role in a /loop-driver. Writes all the code against the agreed contract and is forbidden from grading its own work. Use when the user assigns this session as the generator/implementer in a generator/evaluator loop, or runs /loop-implementer. - 90loop-planner
The planner role in a /loop-driver. Turns a vague human goal into a gradable contract of testable assertions and never touches code. Use when the user assigns this session as the planner in a generator/evaluator loop, or runs /loop-planner. - 85evaluator-contract
Run or compose Spec Kit Evaluator Contract results, inspect provenance and contradictions, and explain recovery outcomes from ordinary-language requests. Use for evaluator composition and reporting, including AEE results. - 100create-evaluator
Use when the user wants to create a Langfuse LLM-as-a-Judge evaluator. Trigger phrases: "create evaluator", "add evaluator", "new evaluation", "set up evaluation criteria", "create judge". Handles prompt composition, schema validation, ID generation, SQL insertion into… - 100gan-style-harness
GAN-inspired Generator-Evaluator agent harness for building high-quality applications autonomously. Based on Shrijan's March 2026 harness design paper. Use when a feature should be built autonomously through generator and evaluator iteration until it clears a quality bar. - 100run-next-sprint
Drive one full sprint of the proof refinement harness: spawn the Planner to write a Sprint Contract, spawn the Generator to execute it, spawn the Evaluator to grade it, and report a one-paragraph status. Use when the user runs /run-next-sprint or when /loop fires this command.