multiple-choice-evaluation
/multiple-choice-evaluation
How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model''s free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B o
- Author
- zjunlp ↗
- Repository
- mechanist ↗
- Licence
- MIT
- Last repository change
- 17.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 17.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model''s free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B o
Triggers from the description
Original author description
How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model''s free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B o
How it works
- The point
- Recipe
- Non-negotiables
- Diagnosing a suspected parser artifact
- Composition
SKILL.md preview
---
name: multiple-choice-evaluation
description: 'How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model''s free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B orientation) and token bias (`A` gets more prior mass than `B`). Triggers: "answer A or B", `[A-D]` regex on generations, any per-choice rate (e.g. `P(chosen)`) computed from a letter parse.'
---
# Multiple-Choice Evaluation (LLM Judge, Not Regex)
## The point
MCQ probes compute their headline metric (accuracy, `P(chosen)`, `Δaccuracy` between arms, per-choice rate) by parsing a letter out of a free-form generation. **Parsing is the measurement** — a parser that mislabels 5–15 % of rows fabricates or destroys the Δ.
Naive case-insensitive letter regex has three symmetric failures:
1. **False positive — letter inside a word.** `re.search(r"[AB]", ...)` matches the first `A/a/B/b` anywhere: `"At the garden…"` → `A`, `"a valid choice…"` → `A`, `"option B refers to…"` before rejection → `B`. Fine-tuned / steered models that stop obeying "answer A or B only" produce exactly these strings.
2. **False negative — over-tight regex.** Tightening to `\banswer[:\s]+([AB])\b` misses `"A."`, `"(A)"`, `"I'd go with A"`, non-English preambles. The trade-off is inescapable inside a regex; disambiguation needs semantics.
3. **Silent coercion.** A two-bucket parser maps refusal / off-topic to whichever letter appears first (usually `A`). A 30 % refusal rate reads as a 30 % A-preference — a phantom effect that survives seeds because the coercion is deterministic.
## Recipe
**1. LLM judge on the full generation.** Send the raw generation plus the question stem with **all** option texts (not only letters — text is what removes the ambiguity a regex can't resolve) to a capable judge; the final per-row verdict must be three-way `CORRECT` / `INCORRECT` / `OTHER`. Refusal / off-topic / multiple / none / format-broken all land in `OTHER` and are **never** coerced into `CORRECT` or `INCORRECT`; report separately or drop from the denominator. A strict regex is fine as a fast pre-filter, but any prose after the match falls back to the judge. Two prompt shapes work — pick one:
**(a) Extract-then-compare.** Judge only extracts which option the model picked (`A` / `B` / … / `OTHER`); a hard-coded step outside the LLM compares against the gold letter. Cleanly separates parsing from scoring, and the extracted letter is auditable on its own.
```
You are extracting a multiple-choice answer from a model's free-form output.
Question and options:
{question_with_options}
Model's free-form answer:
{model_answer}
Which option did the model pick? Reply with EXACTLY ONE of:
A / B / … — model's answer semantically picks that option.
OTHER — refused, off-topic, picked multiple options, unparseable, or matches no option.
Do NOT guess a letter when the answer is a refusal or off-topic.
```
Then downstream: `verdict = "CORRECT" if extracted == gold_letter else ("OTHER" if extracted == "OTHER" else "INCORRECT")`.
**(b) Gold-relative direct judge.** Hand the judge the gold letter as well and ask for `CORRECT` / `INCORRECT` / `OTHER` in one shot. Fewer moving parts, and side-steps the judge's own token bias on `A` vs `B` because it never has to name a letter.
```
You are grading a model's answer to a multiple-choice question.
The gold answer is option {gold_letter}. Question and full option list:
{question_with_options}
Model's free-form answer:
{model_answer}
Reply with EXACTLY ONE of:
CORRECT — model's answer semantically matches option {gold_letter}.
INCORRECT — model's answer semantically picks a different option.
OTHER — refused, off-topic, picked multiple options, unparseable, or matches no option.
Do NOT coerce a refusal or off-topic answer into INCORRECT.
```
Output shape (bare word, JSON field, XML tag) is up to you, as long as the three verdicts are the only legal values.
**2. Control position bias by rotating option order.** Instruction-tuned models carry a ~5–15 pp position bias for the first-listed option that persists after content swap. Run each item in every orientation (A/B swap for binary; a fixed rotation, e.g. 2–4 permutations, for A-D) and report per-orientation as well as averaged. A stable effect appears in every orientation; a sign-flip or > 50 % shrink under rotation is position bias, not the intervention.
**3. Never score with log-prob alone — it's blind to output corruption.** Verbalizer / cloze scoring is contaminated by token bias (`A` gets more prior mass), and — critically — it reads probability off the answer tokens without ever generating text, so a fully broken model (repetition loops, gibberish, refusals, off-language output) still yields a "clean" A-vs-B score you can't distinguish from real answering. Log-prob **cannot** detect output-corruption regressions; always generate on a subset and eyeball the outputs first. If used at all, cross-check against text-gen + judge on a held-out slice; > 5 pp disagreement → keep log-prob as diagnostic only.
## Non-negotiables
- **Three-way `{CORRECT, INCORRECT, OTHER}` verdict required** (whichever prompt shape produced it). Two-way parsers invent effects.
- **All orientations, always** — per-orientation numbers reported (A/B swap for binary; a fixed rotation for A-D).
- **Freeze judge config across arms** (model, prompt, temperature ≤ 0.2). Changing the judge confounds the intervention with the parser.
- **Persist per-row** `(prompt, orientation, raw_generation, judge_verdict)` (plus rationale if collected) so audits don't need re-inference.
- **No judge LLM resource in `task.md` → do NOT self-pick one.** The judge is part of the metric; a silently-chosen judge changes what the headline number means. Log as an **open item** in `EXPERIMENT_RESULTS.md` and block the eval until the user supplies model + base URL + API key (or a local judge path).
## Diagnosing a suspected parser artifact
Any one of these fires ⇒ the effect is un-measured until the parser is fixed:
- **Orientation instability** — Δ changes sign or shrinks > 50 % across orientations ⇒ position-bias-driven.
- **`OTHER` rate rises in the treated arm** ⇒ two-way coercion artifact, not a preference shift. Recompute with `OTHER` excluded.
- **Regex-vs-judge disagreement > 5 % on a 100-row spot-check** ⇒ one of them is systematically wrong (almost always the regex).
## Composition
- **Runs before Phase-5 power / fidelity checks.** An un-audited parser is un-measured, not under-powered — don't tag `suspected_under_power` on a regex-only null; re-run with the judge, then flag `suspected_parser_artifact` if the null moves.
- **Load whenever a plan combines instruction-breaking interventions with an MCQ eval.** fine-tuning (`../finetune-hyperparameter-sweep/`) and mid-range steering (`../steering-coefficient-tuning/`, `../steering-block-selection/`) frequently disable letter-only obedience — the exact regime where regex parsing silently fails.
- **Re-audit on model-family change.** Output-format conventions (`A.` vs `A)` vs `**A**` vs `Answer: A` vs non-English preamble) shift between families and between base / instruct / chat variants; refresh the judge's few-shot exemplars.
Versions
The database has no other skills named multiple-choice-evaluation.
Database snapshot: 13.11.2026 · package version ae55d3288043 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90llm-as-judge
Score open-ended LLM/agent outputs with another LLM as evaluator while controlling for known judge biases. Use when: score open-ended output, llm as judge, evaluation rubric, pairwise comparison eval, judge bias mitigation, calibrate judge against humans. - 100llm-as-judge
Design and validate LLM-as-judge scoring — pairwise vs pointwise, bias correction, anchor calibration, and the cases where a judge is the wrong tool. Use when the user is building an eval, scoring open-ended outputs, or comparing model versions and mentions LLM-as-judge, model… - 100eval-llm-as-judge-system-prompt
Use when configuring or invoking the LLM-as-judge evaluation system that scores legal AI outputs against multiple rubrics. Provides the canonical system prompt template, JSON output schema, ensemble configuration, and bias mitigation rules for automated quality scoring of legal AI responses. - 100agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality". - 100build-llm-judge
Design, calibrate, and bias-audit an LLM-as-judge so its scores track human judgment: named criteria with anchored levels (not 'rate 1-10'), a structured verdict + reason, a pinned judge model/version, a human-agreement calibration on a sample, and checks for position /… - 100llm-judge-rubric-design
Design LLM-as-judge evaluators - decomposed criteria with anchored scales, pointwise versus pairwise format, judge prompt structure, calibration against human labels, and controls for position, verbosity, and self-preference bias. Use when automating quality scoring of model…