Skip to content
Skills Library
AI & agents · Testing

eval-model

/eval-model

Measures the task accuracy of text models served by MAX using standard benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and BABILong. Use when benchmarking a served model, comparing it with model-card or reference scores, verifying that a new MAX model produces correct answers, or running repeatable dataset evaluations against a MAX OpenAI-compatible endpoint.

Repository
skills ↗
Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
10
Scripts
3
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 3 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/_eval_tasks.py, scripts/eval_dataset.py, tests/test_eval_dataset.py

What it does and when to use it

Measures the task accuracy of text models served by MAX using standard benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and BABILong. Use when benchmarking a served model, comparing it with model-card or reference scores, verifying that a new MAX model produces correct answers, or running repeatable dataset evaluations against a MAX OpenAI-compatible endpoint.

Arguments

/eval-model --model <MODEL_ID> --tasks gsm8k [--host localhost]
Original author description

Measures the task accuracy of text models served by MAX using standard benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and BABILong. Use when benchmarking a served model, comparing it with model-card or reference scores, verifying that a new MAX model produces correct answers, or running repeatable dataset evaluations against a MAX OpenAI-compatible endpoint.

How it works

  1. References
  2. Set up the evaluator
  3. Serve the model
  4. Run an evaluation
  5. Interpret the result
  6. Choose tasks
  7. Evaluate reasoning models
  8. Monitor progress
  9. Troubleshoot failures
Original SKILL.md headings · 324 lines, 1,885 words
SKILL.md preview
---
name: eval-model
description: >
  Measures the task accuracy of text models served by MAX using standard
  benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA,
  WinoGrande, and BABILong. Use when benchmarking a served model, comparing it
  with model-card or reference scores, verifying that a new MAX model produces
  correct answers, or running repeatable dataset evaluations against a MAX
  OpenAI-compatible endpoint.
argument-hint: "--model <MODEL_ID> --tasks gsm8k [--host localhost]"
disable-model-invocation: true
compatibility: "Pixi, network access for datasets, and a running MAX endpoint."
---

# Evaluate a model's accuracy on MAX

A model that serves cleanly can still answer benchmark questions wrong. This
skill measures how accurately a text model behind MAX answers standard
datasets — GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and
BABILong. It checks endpoint compatibility before each run, separates serving
failures from wrong answers, and writes reproducible per-task scores you can
compare against a model card.

**Use this skill when** you're benchmarking a served model, comparing it with
model-card or reference scores, verifying that a newly imported MAX model answers
correctly, or running repeatable dataset evaluations against a MAX
OpenAI-compatible endpoint.

**Do not use this skill when** the model doesn't serve yet. Bring the model up
first (`import-model`); if it serves but generates wrong output, chase the
divergence with `debug-model` before benchmarking.

## References

| File | Read when |
|------|-----------|
| [references/examples.md](references/examples.md) | You want more command variants — remote endpoints, artifact paths, long context |
| [references/fast-fail-thresholds.md](references/fast-fail-thresholds.md) | Before changing seed floors or adding a seed task |
| [references/heartbeat-protocol.md](references/heartbeat-protocol.md) | Building a program that consumes the progress stream |

Read the reference for what you're doing, not all of them upfront.

## Set up the evaluator

Resolve the absolute path to the directory containing this `SKILL.md` and save
it as `SKILL_DIR`. Every command below depends on it, so set it from the skill's
real location — not the user's working directory and not the literal placeholder
below. If you were handed the skill path, export that directly; otherwise resolve
it from the path to this file:

```bash
export SKILL_DIR="$(cd "$(dirname /path/to/eval-model/SKILL.md)" && pwd)"
```

Confirm it points at the manifest before continuing, so a wrong value fails here
rather than midway through a run:

```bash
test -f "$SKILL_DIR/pixi.toml" && echo "SKILL_DIR ok: $SKILL_DIR"
```

Install the evaluator environment from its Pixi manifest:

```bash
pixi install --manifest-path "$SKILL_DIR/pixi.toml"
```

This environment holds the evaluator's own dependencies (lm-eval, datasets,
transformers).

Some datasets require internet access, Hugging Face authentication, or license
acceptance. Set `HF_TOKEN` before the run when the dataset requires it.

## Serve the model

The tasks fall into two groups that serve the model differently — decide which
you're running, or serve so you can run both:

- **Generation tasks** (`gsm8k`, `aime`) score a generated answer and run
  against a normally served model.
- **Multiple-choice tasks** (`mmlu`, `hellaswag`, `arc_easy`, `arc_challenge`,
  `winogrande`, `truthfulqa`, `gpqa`) score by prompt log probabilities, which
  MAX only returns when the server starts with `--enable-echo`.

Serve from an environment where MAX is installed.

For generation tasks, start the model normally:

```bash
max serve --model meta-llama/Llama-3.1-8B-Instruct
```

For multiple-choice tasks, add `--enable-echo`:

```bash
max serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-echo
```

You can't toggle echo on a live server. If one is already running without
`--enable-echo`, stop and restart it before evaluating a multiple-choice task —
there's no way to add the capability to a running process. The evaluator's
preflight is the authoritative check: it verifies prompt log probabilities
before downloading datasets and reports a configuration error if they're
unavailable.

If a model's `generation_config.json` sets a non-default `repetition_penalty`
(the Qwen2.5 family ships `1.05`, for example), MAX auto-enables its penalty
sampling path, which is shape-incompatible with echo and crashes the serving
worker at startup with a `repetition_penalty` tensor-shape error. Models that
leave `repetition_penalty` unset (such as Llama-3.1 and Qwen3) don't trigger this
path. If you hit the crash, evaluate a model whose generation config leaves
`repetition_penalty` unset, or restrict the run to generation tasks (`gsm8k`,
`aime`).

## Run an evaluation

Invoke the evaluator through its Pixi manifest from any working directory:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "meta-llama/Llama-3.1-8B-Instruct" \
  --tasks gsm8k
```

For a quick endpoint and scoring check, lower both limits and use the bundled
scorer:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "meta-llama/Llama-3.1-8B-Instruct" \
  --tasks gsm8k \
  --seed-limit 4 \
  --full-limit 8 \
  --direct-http
```

Use `--direct-http` only for connectivity and parsing checks like this one. Its
bundled scorer is 0-shot with a custom prompt, so its numbers run far below the
few-shot chain-of-thought scores on a model card — in one measured run the same
model scored 33% under the direct seed scorer and 82% on the lm-eval full pass.
Don't compare `--direct-http` results with published benchmarks; drop the flag
for any card comparison so the default lm-eval pass runs instead.

For multiple-choice tasks on an aliased served model, pass the Hugging Face
tokenizer ID with `--tokenizer`. This is required, not optional: lm-eval needs a
real tokenizer to compute token boundaries, and without it the tokenizer
defaults to `--model` and lm-eval fails trying to load the alias as a Hugging
Face repo.

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "my-served-alias" \
  --tokenizer "meta-llama/Llama-3.1-8B-Instruct" \
  --tasks mmlu,hellaswag
```

The `--model` value must match an ID returned by `/v1/models`.

`--apply-chat-template` wraps each prompt in the model's chat template. Whether
to use it depends on your comparison baseline, not just on whether the model is
instruction-tuned:

- To reproduce published multiple-choice leaderboard scores (MMLU, HellaSwag,
  ARC, WinoGrande), leave it **off** — those references are measured on raw
  completion prompts. Adding it succeeds but produces numbers that can't be
  compared to the card.
- Use it only when you intend to measure the model as it's actually chatted to,
  and are comparing against a chat-template baseline.

## Interpret the result

The command uses these exit codes:

- `0`: All tasks completed, or every explicit `--target` was met.
- `1`: Setup, dependency, endpoint, dataset, or lm-eval failure.
- `2`: At least one explicit `--target` was missed.
- `3`: A direct seed task scored below its fast-fail floor.

Without `--target`, a valid full run has status `completed`; the evaluator
doesn't impose model-independent accuracy targets. A `completed` or `met` status
means the run finished cleanly — it's not by itself a verdict that the model is
correct or at parity. To turn a run into a pass/fail or a comparison, supply a
baseline yourself.

### Compare against a model card

The evaluator doesn't know any model's expected scores, so a real comparison is
a few explicit steps:

1. Look up the specific metric on the model card or reference, matching the
   variant this tool produces — for generation tasks that's few-shot
   chain-of-thought (for example, Llama-3.1-8B-Instruct reports GSM8K around
   84.5, 8-shot CoT). Note the shot count and whether a chat template was used.
2. Pass that number as `--target`. Be aware `--target` is a one-sided floor: the
   run fails (exit `2`) only when accuracy falls *below* it, so it gates rather
   than measuring a two-sided delta.
3. Read `accuracy` from the task's summary JSON and report the actual difference
   from the reference. Allow for sampling noise — the default full run is 200
   samples per task (or per subtask for grouped tasks), not the full test set,
   so treat small gaps as noise rather than regressions. Raise `--full-limit`
   toward the full dataset when you need a tighter estimate.

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "meta-llama/Llama-3.1-8B-Instruct" \
  --tasks gsm8k,mmlu \
  --target "gsm8k:0.84,mmlu:0.68"
```

Summaries default to `eval-results/<model-name>/`:

```text
eval-results/meta-llama_Llama-3.1-8B-Instruct/
├── eval_summary.json
├── gsm8k_summary.json
└── mmlu_summary.json
```

Use `--output-dir` to choose another location or `--run-id` to change only the
default directory label. The combined summary records the effective CLI
configuration and evaluator dependency versions needed to interpret the run.

## Choose tasks

Task behavior is as follows:

- `gsm8k` and `aime`: Direct-HTTP seed check, followed by an lm-eval full run.
  Add `--direct-http` to use the bundled exact-match scorer for the full run.
- `mmlu`, `hellaswag`, `arc_easy`, `arc_challenge`, `winogrande`, and
  `truthfulqa`: lm-eval full run through `/v1/completions`; require
  `--enable-echo`.
- `gpqa`: lm-eval multiple-choice run through `/v1/completions`; requires
  `--enable-echo`, and the dataset can require Hugging Face access.
- `babilong`: lm-eval chat run; defaults to
  `{"max_seq_lengths": "16k"}` metadata.

Each full run defaults to 200 samples (200 per subtask for grouped tasks such as
`mmlu`; `gpqa` and `babilong` default to 100). This is a spot check, not the
whole test set — raise `--full-limit` when you need a tighter estimate, and
expect small run-to-run variation at the default size. The summary reports the
aggregate number of effective child samples for grouped tasks.

Only the generation tasks (`gsm8k`, `aime`) run the direct seed pass; the
multiple-choice tasks skip straight to the full run, so seeing no seed accuracy
for `mmlu` and friends is expected. The seed's fast-fail floor is a
plumbing gate — it catches an unusable endpoint, a malformed request, or broken
answer extraction, not normal model-quality variation. To judge quality, use
`--target` (see "Compare against a model card"), not the fast-fail floor.

ChartQA isn't registered because stock lm-eval ChartQA prompt formatting isn't
compatible with MAX image-token injection. Add a validated MAX-specific task
configuration before enabling it.

Print the effective task configuration without installing MAX or starting a
server:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" validate
```

## Evaluate reasoning models

For a MAX model whose chat template accepts `reasoning_effort`, pass one of
`low`, `medium`, `high`, or `xhigh`:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "<MODEL_ID>" \
  --tasks gsm8k,aime \
  --reasoning-effort low
```

The evaluator sends this value through
`chat_template_kwargs={"reasoning_effort": ...}` and automatically selects the
direct-HTTP full scorer for supported generation tasks. The preflight request
fails clearly if the endpoint doesn't accept the field.

## Monitor progress

Heartbeat records go to stdout; human-readable logs go to stderr. To archive
them, capture each stream to its own file:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "<MODEL_ID>" \
  --tasks gsm8k \
  > heartbeats.jsonl \
  2> eval.log
```

To watch liveness *while* the run proceeds, keep stdout on a pipe rather than a
plain file, so a consumer can read heartbeats as they arrive — the example
consumer in the heartbeat reference can't tail a file redirect alone:

```bash
pixi run --manifest-path "$SKILL_DIR/pixi.toml" eval \
  --model "<MODEL_ID>" --tasks gsm8k \
  2> eval.log | tee heartbeats.jsonl | python your_consumer.py
```

Treat a missing heartbeat for twice `--heartbeat-interval` as a possible stall.
For reasoning models, a single generation can legitimately exceed the default
30-second interval, so raise `--heartbeat-interval` (and with it the stall
threshold) to avoid false alarms on slow-but-healthy runs. The evaluator emits
`done` only after writing the corresponding task summary.

## Troubleshoot failures

Use these checks:

- Connection or model error: Run `curl http://localhost:8000/v1/models` and
  pass an exact returned model ID.
- Missing prompt logprobs: Restart MAX with `--enable-echo` (it can't be
  enabled on a running server); confirm the selected runtime supports log
  probabilities.
- Serving worker crashes on startup with echo and a `repetition_penalty`
  tensor-shape error: The served model's `generation_config.json` sets a
  non-default `repetition_penalty` (common in Qwen2.5), so MAX auto-enables a
  penalty path that echo doesn't support. Use a model whose generation config
  leaves `repetition_penalty` unset (Llama-3.1, Qwen3) for multiple-choice tasks,
  or restrict the run to generation tasks (`gsm8k`, `aime`).
- `No module named 'pydantic'` from `max serve`: The serving environment has the
  slim `max` package but not `max-pipelines`. Install `max-pipelines`.
- Missing package: Run
  `pixi install --manifest-path "$SKILL_DIR/pixi.toml"` again.
- Dataset access error: Authenticate with Hugging Face and accept the dataset's
  terms.
- lm-eval task error: List tasks for the installed lm-eval version with
  `pixi run --manifest-path "$SKILL_DIR/pixi.toml" lm-eval ls tasks`, then print
  the effective task configuration with
  `pixi run --manifest-path "$SKILL_DIR/pixi.toml" validate` (the `validate` task
  runs the script's `--validate-thresholds` mode).
- Out-of-memory or context-length error: Reduce `--num-concurrent`,
  `--full-limit`, or the model's served context length as appropriate.
Files in this package (10)
  • .gitattributes 1 KB
  • .gitignore 1 KB
  • SKILL.md 14 KB
  • pixi.toml 1 KB
  • references/examples.md 4 KB
  • references/fast-fail-thresholds.md 2 KB
  • references/heartbeat-protocol.md 4 KB
  • scripts/_eval_tasks.py 25 KB
  • scripts/eval_dataset.py 35 KB
  • tests/test_eval_dataset.py 10 KB

Versions

The database has no other skills named eval-model.

Database snapshot: 13.11.2026 · package version acca510bd22c (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 84lm-evaluation-harness
    Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs.…
  • 100evaluating-llms-harness
    Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs.…
  • 100evaluating-llms-harness
    Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs.…
  • 100serve-model
    Serve a model with MAX's `max serve` command: set up the environment (pixi or uv with the max-nightly conda channel / nightly wheel index), point the server at a Hugging Face repo or local checkpoint, target a custom architecture with `--custom-architectures`, and pick the right…
  • 95openai-compatible-endpoint-pack
    Use when a Codex task needs to configure, diagnose, or migrate an AI coding project to an OpenAI-compatible API endpoint without leaking credentials.
  • 100huggingface-community-evals
    Runs benchmark evaluations of Hugging Face Hub models on local GPUs with lighteval or inspect-ai (GSM8K, HellaSwag, MMLU and similar) and picks the backend (vLLM, Transformers or accelerate) with exact commands. Use when the user wants to evaluate or benchmark an HF model…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.