benchmark
/benchmark
Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движк
- Author
- serejaris ↗
- Repository
- personal-corp-os ↗
- Licence
- MIT
- Last repository change
- 21.10.2026
Assessment and security
- Files in this package
- 12
- Scripts
- 7
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 21.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 7 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: examples/adapter_example.py, scripts/asr.py, scripts/claude_step.py, scripts/common.py, scripts/llm.py, scripts/report.py, scripts/score.py
What it does and when to use it
Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движк
Triggers from the description
Original author description
Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движк
How it works
- When to use
- Rules
- Steps
- Engines and providers
- Metrics
- Bias checklist (write it into the method note)
- Container
- Files
SKILL.md preview
--- name: benchmark description: >- Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движки", "сколько стоит час записи", "какую модель взять для пайплайна", "benchmark the pipeline", "compare ASR engines". --- # Benchmark Measure a pipeline on your own input before you pick an engine. One task file lists the variants; the scripts run them one by one in an isolated container and write numbers you can defend: time, cost per hour of input with a dated price source, quality against a reference. Shape of a pipeline this skill knows: `input (recording) → ASR engine → transcript → author model (notes, chapters) → code checks → judge model` Each arrow is a slot. A variant fills one slot and keeps the rest fixed. ## When to use - «ElevenLabs, Whisper or a Russian engine: what does an hour of our meetings cost and who makes fewer mistakes»; - «move the notes step from Claude to GLM: does quality hold»; - before a pipeline goes unattended: pick the critic that finds the most real problems. Not for: load testing, latency SLOs of a live service, model evals on public datasets. ## Rules 1. **Isolation.** Run in a dedicated container (LXC/VM), not on a laptop and not next to production: the timing is clean and the keys live only there. Egress: provider APIs; private network closed except the services the task needs (for example a self-hosted ASR). 2. **Keys only from env.** The container gets a root-only env file; scripts read variable names from the task and never print values. Error bodies are redacted (some providers echo the key back). 3. **Same harness for every model.** LLM steps run through `claude -p` (Claude Code headless) against Anthropic-compatible endpoints; only the model changes. Tools, prompt, permissions stay the same. 4. **Judge is never the author's model.** `llm.py` refuses such a variant. 5. **No answer leaks.** The author must not see the published result of the same input (a «form reference» that is the answer itself). Give it a sibling of the same kind. 6. **Prices carry source and date.** `prices.json`: every item has `source` and `checked`; FX from the central bank of the day. A number without a source does not go into the report. 7. **What was not run is a row too.** Missing key, no balance, provider closed to new clients: write the reason and the list price, do not drop the row. 8. **Private input stays private.** Transcripts and recordings go to a private repo; the analytics page shows only aggregate numbers. ## Steps 1. **Task file** (`task.example.json`): input audio, language, terms list, reference transcript, ASR variants, LLM variants with judges, `not_run` rows with reasons. 2. **Container.** Create it by your infra rules (the section «Container» below). Install `ffmpeg`, Python venv with `httpx jiwer faster-whisper`, Node and `@anthropic-ai/claude-code`. Put keys into `/etc/bench.env` (root 600) through a pipe from your secret store. 3. **ASR:** `python3 scripts/asr.py --task task.json --out runs/<name>` (engines run sequentially; ElevenLabs credits are read before and after). 4. **LLM:** write an adapter for your pipeline (contract in `scripts/llm.py`), then `python3 scripts/llm.py --task task.json --out runs/<name> --stage author` on the container and `--stage judge` where the judge's key lives (a subscription login on another machine is fine: the judge only reads files). 5. **Score:** `python3 scripts/score.py --task task.json --out runs/<name> --prices prices.json`. 6. **Page:** `python3 scripts/report.py --results runs/<name>/results.json --out report.html --notes notes.md`. 7. **Method note** next to the results: input, reference origin, what was not run and why, known biases. ## Engines and providers | Slot | Engine key | Tested | Needs | |---|---|---|---| | ASR | `elevenlabs` (Scribe, optional keyterms) | yes | `ELEVENLABS_API_KEY` | | ASR | `openai_compat` (`/v1/audio/transcriptions`: self-hosted GigaAM, speaches, cloud) | yes | server URL, optional key | | ASR | `faster_whisper` (CPU, int8) | yes | model download once | | ASR | `openrouter_audio` (chat with `input_audio`, chunked) | request path only | `OPENROUTER_API_KEY` with balance | | LLM | `zai` (GLM via `api.z.ai/api/anthropic`) | yes | `ZAI_API_KEY` | | LLM | `anthropic-local` (the login on this machine) | yes | Claude Code logged in | | LLM | `openrouter`, `deepseek`, `anthropic-api`, `anthropic-oauth` | not yet | key with balance | Yandex SpeechKit and SaluteSpeech have no adapter yet: add one to `ENGINES` in `asr.py` (input: audio path, output: text, raw response, usage) when you have a key. ## Metrics ASR (against the reference, after lower case, `ё→е`, punctuation and speaker tags removed): - `wer`, `cer`: word and character error rate; - `wer_termnorm`: WER after known term distortions are fixed in both texts; - `term_accuracy`: share of term mentions written canonically, canon / (canon + known distortions), no reference needed; - `term_recall`: canonical term mentions vs. the reference; - `asr_pairwise_wer`: engines against each other, independent of the reference; - `rtf`: wall time / audio time. LLM: code checks before and after the fix round, fix rounds, judges' verdicts with critical / major / minor counts, tokens, turns, wall time. Cost per hour of input: list price (`api_price`), tokens × list price (`tokens`), provider-reported cost (`reported`), or the CPU-seconds share of your server's month (`server_share`) for self-hosted engines. ## Bias checklist (write it into the method note) - Where the reference came from. A reference made by engine X favours X in WER; say so and read `term_accuracy` and pairwise WER next to it. - Numbers: engines write «65» or «шестьдесят пять»; WER counts both as errors. - Self-hosted time depends on the host's load at run time; note CPU threads and neighbours. - One recording is one sample. Treat differences under a couple of WER points as noise. ## Container The skill does not create infrastructure. Follow your infra rules; minimum: - a separate container for benchmarks, not a production or user container; - no inbound exposure; no SSH if the host can exec into it; - outbound: internet for APIs; private networks closed except named services; - keys in a root-only env file, services started with `EnvironmentFile=`, run as an unprivileged user; - CPU weight below production neighbours (for example `cpuunits` lower than default). ## Files - `scripts/asr.py`, `scripts/llm.py`, `scripts/claude_step.py`, `scripts/score.py`, `scripts/report.py`, `scripts/common.py` - `task.example.json`: a task with every engine type - `prices.example.json`: price table with sources, checked 2026-09-30 - `examples/adapter_example.py`: a minimal LLM adapter (author → checks → one fix → judge); run end to end with GLM-5.3-Flash as author and GLM-5.3 as judge
Files in this package (12)
- README.md 2 KB
- README.ru.md 3 KB
- SKILL.md 7 KB
- prices.example.json 3 KB
- task.example.json 2 KB
- examples/adapter_example.py 5 KB
- scripts/asr.py 11 KB
- scripts/claude_step.py 6 KB
- scripts/common.py 2 KB
- scripts/llm.py 4 KB
- scripts/report.py 10 KB
- scripts/score.py 15 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · serejaris/personal-corp-os | 100 | MIT |
| workerstack/repo | 100 | MIT |
| YuDefine/nuxt-supabase-starter | 100 | MIT |
| hoangsonww/Claude-Code-Agent-Monitor | 100 | MIT |
| RainingDaemons/agents | 100 | MIT |
| wwylon/devkit | 100 | MIT |
| pekral/cursor-rules | 100 | MIT |
| peabrane/peapods | 100 | MIT |
| nguyenyou/scalex | 100 | MIT |
Database snapshot: 13.11.2026 · package version 9a13adb61940 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100benchmark-models
Cross-model benchmark. Runs one prompt or skill through Claude, GPT (Codex CLI), and Gemini side by side and compares latency, tokens, cost, tool calls, and optionally output quality via an Anthropic-API judge. Reports per-metric leaders with data, and grades a single run per… - 90generative-model-evaluation
Use when choosing among image, video, speech or other generative models for production. Benchmark creative AI models for quality, latency, cost, controls and policy fit using blind tests. Do not use when no model comparison or model-dependent decision exists. - 100benchmark-models
Cross-model benchmark. Runs one prompt or skill through Claude, GPT (Codex CLI), and Gemini side by side and compares latency, tokens, cost, tool calls, and optionally output quality via an Anthropic-API judge. Reports per-metric leaders with data, and grades a single run per… - 100benchmark-models
Compare AI models, reasoning-effort levels, providers, serving tiers or agent harnesses on quality, latency and cost for any LLM call site in the repository, on any model provider or OpenAI-compatible vendor. Use when asked to benchmark, A/B, evaluate or migrate a model, pick a… - 100scenario-model-comparison
Use when comparing Scenario models on the same brief through MCP: a bake-off with shared prompts and inputs, cost, generation time and quality measured side by side, choosing a default model, testing a new release against the incumbent, or finding which model handles a style,… - 100vibe-voice
Run Microsoft's VibeVoice, open-source voice AI for long-form speech recognition with speaker labels and timestamps, plus real-time text-to-speech. Use when: transcribing meetings or podcasts with diarization, adding hotwords for domain terms, streaming speech recognition,…