Skip to content
Skills Library
AI & agents · Testing

benchmark-model

/benchmark-model

Benchmark a model served on MAX with the `max benchmark` command: measure throughput (tokens/sec), latency (TTFT, TPOT, inter-token latency), and GPU utilization by driving load against a running `max serve` endpoint. Use this whenever the user wants to benchmark, load-test, or measure the performance of a MAX model, get tokens-per-second / TTFT / TPOT numbers, run a concurrency or request-rate sweep, compare latency vs throughput, size a deployment, or produce benchmark JSON, even if they don't say "benchmark" by name. Also use when a `max benchmark` run fails to connect or reports zero/garba

Repository
skills ↗
Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
4
Scripts
0
HTML comments
0
URLs in SKILL.md
4
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Benchmark a model served on MAX with the `max benchmark` command: measure throughput (tokens/sec), latency (TTFT, TPOT, inter-token latency), and GPU utilization by driving load against a running `max serve` endpoint. Use this whenever the user wants to benchmark, load-test, or measure the performance of a MAX model, get tokens-per-second / TTFT / TPOT numbers, run a concurrency or request-rate sweep, compare latency vs throughput, size a deployment, or produce benchmark JSON, even if they don't say "benchmark" by name. Also use when a `max benchmark` run fails to connect or reports zero/garba

Triggers from the description

t say

Arguments

/benchmark-model [what to measure, for example 'single-request latency' or 'peak throughput']
Original author description

Benchmark a model served on MAX with the `max benchmark` command: measure throughput (tokens/sec), latency (TTFT, TPOT, inter-token latency), and GPU utilization by driving load against a running `max serve` endpoint. Use this whenever the user wants to benchmark, load-test, or measure the performance of a MAX model, get tokens-per-second / TTFT / TPOT numbers, run a concurrency or request-rate sweep, compare latency vs throughput, size a deployment, or produce benchmark JSON, even if they don't say "benchmark" by name. Also use when a `max benchmark` run fails to connect or reports zero/garba

How it works

  1. References
  2. 1. Check the server and read its model name
  3. 2. Pick the workload for your question
  4. 3. Run it, save results, add GPU stats
  5. 4. Read the metrics
  6. Troubleshooting
Original SKILL.md headings · 158 lines, 1,042 words
SKILL.md preview
---
name: benchmark-model
description: >
  Benchmark a model served on MAX with the `max benchmark` command: measure
  throughput (tokens/sec), latency (TTFT, TPOT, inter-token latency), and GPU
  utilization by driving load against a running `max serve` endpoint. Use this
  whenever the user wants to benchmark, load-test, or measure the performance of
  a MAX model, get tokens-per-second / TTFT / TPOT numbers, run a concurrency or
  request-rate sweep, compare latency vs throughput, size a deployment, or
  produce benchmark JSON, even if they don't say "benchmark" by name. Also use
  when a `max benchmark` run fails to connect or reports zero/garbage numbers.
compatibility: Requires a pip or pixi MAX install and a running `max serve` endpoint; GPU stats (`--collect-gpu-stats`) need the benchmark to run on the same NVIDIA machine as the server.
argument-hint: "[what to measure, for example 'single-request latency' or 'peak throughput']"
---

# Benchmark a model on MAX

`max benchmark` measures a **running** model server. It's a load generator: it
sends inference requests to a live `max serve` endpoint, times them, and reports
throughput (tokens/sec) and latency (TTFT, TPOT, inter-token latency). A
benchmark combines two things: a server under test, and a workload that matches
the question you're asking.

Every run reports both throughput and latency, so there's no mode to select.
Decide what you want to learn first, then pick the workload that measures it.
Single-stream latency and peak throughput come from different workloads, so the
workload you choose *is* the measurement. Get that right and a clean number
falls out.

Use this skill when you want a performance number for a model on MAX:
tokens/sec, TTFT / TPOT, a concurrency or request-rate sweep, a
latency-vs-throughput tradeoff, or deployment sizing.

Don't use this skill when no server is running yet. `max benchmark` is a client,
so start a server with the `serve-model` skill first. To find out *where*
inference time goes at the kernel level, use `profile-model`. To check *whether
the output is correct*, treat it as a parity task (`import-model`, then
`debug-model`) rather than a benchmark.

This skill works anywhere MAX is installed (pip or pixi). Add `pixi run` in a
pixi project.

## References

The following table lists the reference files and when to read each one:

| File | Read when |
|------|-----------|
| [references/flags.md](references/flags.md) | Choosing any flag or dataset beyond the ones below |
| [references/metrics.md](references/metrics.md) | Turning the throughput and latency numbers into a conclusion |
| [references/troubleshooting.md](references/troubleshooting.md) | A run won't connect, requests fail, or the numbers look wrong |

Read the reference for what you're doing, not all of them upfront.

## 1. Check the server and read its model name

```bash
curl -s http://localhost:8000/v1/health     # 200 = ready
curl -s http://localhost:8000/v1/models      # note the served model name
```

Both checks matter before you spend a run:

- The benchmark's `--model` must equal the server's `--served-model-name`
  exactly, or every request fails. Take the value from `/v1/models` rather than
  guessing it.
- If that served name is an *alias* rather than a Hugging Face ID (no `/` in
  it), `--tokenizer` defaults to it and can't resolve, and the run dies with
  "not a valid model identifier." Pass the model's real Hugging Face ID as
  `--tokenizer`.

If nothing is serving, start a server with `max serve` (for a custom
architecture, use the `serve-model` skill). One more server setting matters:
`--max-batch-size` caps real concurrency. A sweep to `--max-concurrency 32`
against a server started with `--max-batch-size 1` queues requests instead of
batching them, so raise the server's batch size to match the sweep or the
high-concurrency points mean nothing.

Wait for `Server ready` *then* benchmark. Benchmarking during compile or warmup
produces garbage first-token times.

## 2. Pick the workload for your question

The workload is the measurement. Match it to what you want to learn:

| What you want to know | Workload |
|---|---|
| Best-case single-request latency (TTFT, TPOT) | `--max-concurrency 1`, fixed `random` shape, small `--num-prompts` |
| Peak throughput and where latency degrades | `--max-concurrency 1,2,4,8,16,32` sweep, more prompts |
| Performance under a realistic mix | `--dataset-name sharegpt`, moderate concurrency |
| Behavior at a target load | `--request-rate 1,2,4,8` sweep (requests/sec) |

A sweep answers the first two rows at once: the concurrency-1 point is the
best-case latency number, and the peak across the sweep is the throughput
number. Reach for a dedicated concurrency-1 run when you only want the latency
figure and don't want to pay for the rest of the curve.

The key knobs are the following (`references/flags.md` has the full catalog):

- `--dataset-name`: pick `random` (synthetic, shape it with
  `--random-input-len` and `--random-output-len`), `sharegpt` (real chat), or
  `arxiv-summarization` (long context). `random` works best for clean,
  reproducible micro-measurements.
- `--max-concurrency` and `--request-rate`: take a single value or a
  comma-separated *sweep* (`1,2,4,8`). A sweep is how you find the throughput
  knee.
- `--endpoint`: use `/v1/completions` for base LMs, which need no chat template,
  or `/v1/chat/completions` for instruct and chat models, which must have a chat
  template or the requests return 400.
- `--max-output-len`: sets the decode length, which dominates how long the run
  takes.
- `--num-prompts`: required for single-turn runs.

## 3. Run it, save results, add GPU stats

For best-case single-request latency, pin concurrency to 1 and keep the shape
fixed:

```bash
pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 128 --random-output-len 128 \
  --max-output-len 128 --num-prompts 32 --max-concurrency 1 \
  --result-filename results/latency.json --collect-gpu-stats
```

For peak throughput and the latency knee, sweep concurrency and send more
prompts:

```bash
pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 512 --random-output-len 128 \
  --max-output-len 128 --num-prompts 200 \
  --max-concurrency 1,2,4,8,16,32 \
  --result-filename results/throughput.json --collect-gpu-stats
```

Note these three things about saving and instrumenting a run:

- `--result-filename`: writes metrics to JSON and creates the directories it
  needs. Set it whenever you want to track or compare runs; without it, MAX
  saves nothing. `--metadata key=value` stamps the JSON, for example
  `--metadata tp=1 gpu=b200`. A sweep also drops a `results-<N>-median.json`
  per step under `--log-dir`.
- `--collect-gpu-stats`: adds GPU utilization and peak memory. This works only
  when the benchmark runs on the **same machine** as the server (NVIDIA).
- For version-controlled configs, put options under a `benchmark_config:` key in
  a YAML file and pass `--config-file file.yaml`. Keys use `snake_case`, and CLI
  flags override the file.

## 4. Read the metrics

The run prints throughput and latency, and a sweep prints one row per point. The
headline numbers are the following:

- Output token throughput (tok/s): the main throughput number.
- TTFT (time to first token): prefill responsiveness. Watch p50 and p99.
- TPOT and ITL (time per output token and inter-token latency): decode speed.
- GPU utilization and peak memory: reported with `--collect-gpu-stats`.

For how to turn these numbers into a conclusion, and the latency-vs-throughput
tradeoff a sweep reveals, see `references/metrics.md`.

## Troubleshooting

Match the symptom against `references/troubleshooting.md`, which covers
connection failures, model-name mismatches, tokenizer-alias errors,
chat-template 400s, flat throughput from a batch-size cap, and warmup-skewed
first-token times. Confirm that `curl /v1/health` returns 200 before you check
anything else.
Files in this package (4)
  • SKILL.md 8 KB
  • references/flags.md 6 KB
  • references/metrics.md 4 KB
  • references/troubleshooting.md 2 KB

Versions

The database has no other skills named benchmark-model.

Database snapshot: 13.11.2026 · package version 8ffc660963e0 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100vllm-xpu-bench
    Benchmark a **running vLLM-XPU OpenAI-compatible server** on an Intel GPU using `vllm bench`. Measures TTFT (time-to-first-token), TPOT (time-per-output-token), ITL (inter-token latency), end-to-end latency, and throughput under concurrency. Covers online (`vllm bench serve`)…
  • 100llm-speedtest
    Benchmark any OpenAI-compatible LLM API endpoint (gateways, routers, self-hosted servers, cloud providers) for availability, latency (TTFT, tokens/s), load capacity, capabilities and answer quality, then rank models and families and write a report. Use when the user asks which…
  • 100evaluate-model-serving-aks
    Evaluate Model Serving AKS — benchmark inference throughput (tokens/sec), latency (TTFT, TPS), GPU utilization, pod health, scaling behavior. Use when: evaluate, benchmark, load test.
  • 90latency-budgeting
    Budget and engineer latency for an LLM agent — TTFT, tokens-per-second, tool round-trips, parallelism, streaming. Use when the user is building a user-facing or real-time agent and mentions latency, p50, p95, p99, TTFT, streaming, throughput, time-to-first-token, slow agent, or…
  • 100arkcli-models
    arkcli 模型查询与基础模型服务激活能力:列出、搜索、获取火山公共基础模型详情,以及用户明确要求的开通/激活模型服务(`arkcli models activate`);Volc 还支持 TTFT/TPOT 性能排名、延迟趋势和输入长度对比。激活已有基础模型服务不等于部署/创建 Endpoint,不得转成 `+deploy`。优先使用产品命令 `arkcli models ...`,而不是直接调用 Raw API。反触发:用户的最终目标是创建 / 部署 Endpoint 时,本 skill 承担有界的只读候选查询,owning skill…
  • 100eval-surfer
    Drive AI application evaluations using the EvalSurfer skill-first workflow. Use when creating AI eval rubrics, reviewing RAG outputs, checking agent tool use, assessing safety, or calculating operational metrics like latency, TTFT, inter-token latency, throughput (tokens per…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.