Skip to content
Skills Library
AI & agents · Testing

benchmark-model-runtime

/benchmark-model-runtime

Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.

Repository
socket ↗
Licence
Apache-2.0
Last repository change
26.08.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.

Original author description

Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.

How it works

  1. Define A Fair Workload
  2. Workflow
  3. Apple Runtime Checks
  4. References
Original SKILL.md headings · 30 lines, 239 words
SKILL.md preview
---
name: benchmark-model-runtime
description: Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.
---

# Benchmark Model Runtime

## Define A Fair Workload

Pin the exact model artifact, tokenizer/template, runtime and version, device and OS, precision, cache policy, batch size, prompt-length buckets, generated-token target, sampling settings, and measurement tool. Compare numerical or behavioral parity before performance.

## Workflow

1. Verify each artifact produces acceptable outputs on the same small parity set.
2. Separate cold load, warm load, prompt processing, time to first token, decode throughput, and end-to-end latency.
3. Measure peak and steady memory; include model, cache, runtime, and process overhead consistently.
4. Stabilize device power, charging, background load, and thermal state. Record deviations instead of silently rerunning only slow samples.
5. Warm up separately, then run enough measured repetitions to report median and tail percentiles.
6. Sweep representative prompt lengths, output lengths, and batch/concurrency levels.
7. Record failures, fallback execution, recompilation, memory pressure, and thermal throttling.
8. Measure energy with an appropriate system tool when the decision depends on battery or sustained deployment.
9. Retain raw samples and summarize them with units, sample counts, and uncertainty.

## Apple Runtime Checks

- Confirm delegated operator coverage and fallback behavior rather than assuming the named backend ran the whole graph.
- Distinguish compilation/conversion time from load and inference time.
- For stateful generation, include key-value cache initialization, update, and memory growth.
- Treat ExecuTorch MLX results as revision-specific while the upstream backend remains experimental.

## References

Use `references/runtime-benchmarking.md` for metric definitions and reporting requirements.
Files in this package (3)
  • SKILL.md 2 KB
  • agents/openai.yaml 1 KB
  • references/runtime-benchmarking.md 1 KB

Versions

The database has no other skills named benchmark-model-runtime.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version b34a89e7a038 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90choose-apple-model-runtime
    Compare and select Core AI, Core ML, MLX, MLX Swift, MLX LM, ExecuTorch Apple delegates, or Foundation Models. Use when an Apple model workflow needs a runtime choice and implementation handoff.
  • 100binary-size
    Analyze and reduce ExecuTorch binary size. Use when investigating binary size, running size tests, or optimizing the runtime for size-constrained deployments.
  • 100edge-ai
    Use this skill when the user says 'edge AI', 'on-device inference', 'TensorRT', 'ONNX Runtime', 'TFLite', 'LiteRT', 'TinyML', 'quantization int8', 'Jetson', 'Coral', 'NPU', 'ExecuTorch', 'model on microcontroller'. Deploys trained models to edge hardware: export path (PyTorch/TF…
  • 100executorch-kb
    Search the ExecuTorch tribal knowledge base covering QNN, XNNPACK, Vulkan, CoreML, Arm, and Cadence backends, quantization recipes, export pitfalls, runtime errors, and SoC compatibility. Use when debugging ExecuTorch errors, choosing quantization configs, checking backend op…
  • 100pytorch-model-trainer
    Build pytorch model trainer operations. Auto-activating skill for ML Training. Triggers on: pytorch model trainer, pytorch model trainer Part of the ML Training skill category. Use when working with pytorch model trainer functionality. Trigger with phrases like "pytorch model…
  • 100mlenergy-data
    Answer natural-language questions about The ML.ENERGY Benchmark (LLM and diffusion inference energy/latency/throughput measurement data on GPUs) using the `mlenergy-data` Python toolkit. Use when the user asks about energy per token, energy per image/video, throughput, ITL…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.