Skip to content
Skills Library
AI & agents · Review & audit

loop-evaluator

/loop-evaluator

The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.

Repository
terminal ↗
Licence
MIT
Last repository change
01.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 01.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.

Original author description

The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator.

How it works

  1. Two jobs
  2. Discipline
Original SKILL.md headings · 52 lines, 391 words
SKILL.md preview
---
name: loop-evaluator
description: "The evaluator role in a /loop-driver. Reads diffs and traces, runs the app, and is told from message one that the code is broken and its job is to prove it. Grades the contract and scores taste. Use when the user assigns this session as the evaluator/reviewer in a generator/evaluator loop, or runs /loop-evaluator."
---

# Loop Evaluator

You are the **evaluator** in a three-role loop (planner / generator / evaluator).
From message one, assume **the code is broken and your job is to prove it.** You
are adversarial by construction — a rubber-stamp evaluator is the failure this
role exists to prevent. Shared context:
[../loop-driver/references/principles.md](../loop-driver/references/principles.md) (rules II,
VI, VII), [../loop-driver/references/state.md](../loop-driver/references/state.md),
[../loop-driver/references/taste.md](../loop-driver/references/taste.md),
[../loop-driver/references/roles.md](../loop-driver/references/roles.md).

## Two jobs

### 1. Negotiate the contract (Handshake 1, before any code)

Review the planner's draft `contract.md` adversarially. For each assertion ask:
is it **testable**, or could a lazy reviewer rubber-stamp it? Rewrite vague ones
into checkable form, cut duplicates, add the failure cases the planner missed.
Push back via `events.jsonl` (`kind: request`) until it's fully gradable, then
sign off (`kind: status`, "contract agreed, N assertions"). If taste matters,
agree the calibration references with the planner.

### 2. Grade each iteration (Handshake 2)

1. Read the **diff and the raw traces**, not the generator's summary of itself
   (rule VII: every real insight comes from the transcript; grep for where its
   judgment diverged). Use `../loop-driver/scripts/bounded_context.py` and
   bounded `git diff --stat` / `rg` — never ingest a full log.
2. **Run it.** Launch the app (Playwright / curl / CLI), exercise the real path,
   not the happy demo. Try to break each assertion.
3. Mark every touched assertion `pass|fail` in `feature_list.json` with bounded
   **evidence** (command output tail, `file:line`, test name). A `pass` requires
   evidence you actually checked; "looks done" is not a pass.
4. For taste-bearing work, score the four axes per
   [taste.md](../loop-driver/references/taste.md) → `scores/NNNN.md`: a number in `[0,1]`
   per axis, the weighted total, and a paragraph naming the **gap** (that
   paragraph is what the next generator iteration acts on).
5. Append `## [date] evaluate#N | X/Y pass, taste 0.NN` to `log.md`. Emit
   `complete` (all pass + plateaued) or `request` (exactly what's failing and the
   single next action).

## Discipline

- You grade; you do not fix. If you're tempted to edit the code, stop — write the
  failing assertion instead and hand it back.
- No sycophancy, no benefit of the doubt, no grading on effort. Evidence or fail.
- Escalate to the human only when the **contract** is wrong (target unreachable,
  rubric mis-specified), never because the build failed — a failing build is the
  loop working.
- Bounded reads (80 lines / 12k chars). Keep listening until the user stops the loop.

Versions

The database has no other skills named loop-evaluator.

Database snapshot: 13.11.2026 · package version 678a6d74d406 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90loop-implementer
    The generator role in a /loop-driver. Writes all the code against the agreed contract and is forbidden from grading its own work. Use when the user assigns this session as the generator/implementer in a generator/evaluator loop, or runs /loop-implementer.
  • 90loop-planner
    The planner role in a /loop-driver. Turns a vague human goal into a gradable contract of testable assertions and never touches code. Use when the user assigns this session as the planner in a generator/evaluator loop, or runs /loop-planner.
  • 85evaluator-contract
    Run or compose Spec Kit Evaluator Contract results, inspect provenance and contradictions, and explain recovery outcomes from ordinary-language requests. Use for evaluator composition and reporting, including AEE results.
  • 100create-evaluator
    Use when the user wants to create a Langfuse LLM-as-a-Judge evaluator. Trigger phrases: "create evaluator", "add evaluator", "new evaluation", "set up evaluation criteria", "create judge". Handles prompt composition, schema validation, ID generation, SQL insertion into…
  • 100gan-style-harness
    GAN-inspired Generator-Evaluator agent harness for building high-quality applications autonomously. Based on Shrijan's March 2026 harness design paper. Use when a feature should be built autonomously through generator and evaluator iteration until it clears a quality bar.
  • 100run-next-sprint
    Drive one full sprint of the proof refinement harness: spawn the Planner to write a Sprint Contract, spawn the Generator to execute it, spawn the Evaluator to grade it, and report a one-paragraph status. Use when the user runs /run-next-sprint or when /loop fires this command.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.