benchmark-model-runtime
/benchmark-model-runtime
Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.
- Author
- gaelic-ghost ↗
- Repository
- socket ↗
- Licence
- Apache-2.0
- Last repository change
- 26.08.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.08.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.
Original author description
Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.
How it works
- Define A Fair Workload
- Workflow
- Apple Runtime Checks
- References
SKILL.md preview
--- name: benchmark-model-runtime description: Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging. --- # Benchmark Model Runtime ## Define A Fair Workload Pin the exact model artifact, tokenizer/template, runtime and version, device and OS, precision, cache policy, batch size, prompt-length buckets, generated-token target, sampling settings, and measurement tool. Compare numerical or behavioral parity before performance. ## Workflow 1. Verify each artifact produces acceptable outputs on the same small parity set. 2. Separate cold load, warm load, prompt processing, time to first token, decode throughput, and end-to-end latency. 3. Measure peak and steady memory; include model, cache, runtime, and process overhead consistently. 4. Stabilize device power, charging, background load, and thermal state. Record deviations instead of silently rerunning only slow samples. 5. Warm up separately, then run enough measured repetitions to report median and tail percentiles. 6. Sweep representative prompt lengths, output lengths, and batch/concurrency levels. 7. Record failures, fallback execution, recompilation, memory pressure, and thermal throttling. 8. Measure energy with an appropriate system tool when the decision depends on battery or sustained deployment. 9. Retain raw samples and summarize them with units, sample counts, and uncertainty. ## Apple Runtime Checks - Confirm delegated operator coverage and fallback behavior rather than assuming the named backend ran the whole graph. - Distinguish compilation/conversion time from load and inference time. - For stateful generation, include key-value cache initialization, update, and memory growth. - Treat ExecuTorch MLX results as revision-specific while the upstream backend remains experimental. ## References Use `references/runtime-benchmarking.md` for metric definitions and reporting requirements.
Files in this package (3)
- SKILL.md 2 KB
- agents/openai.yaml 1 KB
- references/runtime-benchmarking.md 1 KB
Versions
The database has no other skills named benchmark-model-runtime.
Identical content appears in 1 other GitHub folders. We show one copy with the highest score.
Database snapshot: 13.11.2026 · package version b34a89e7a038 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90choose-apple-model-runtime
Compare and select Core AI, Core ML, MLX, MLX Swift, MLX LM, ExecuTorch Apple delegates, or Foundation Models. Use when an Apple model workflow needs a runtime choice and implementation handoff. - 100binary-size
Analyze and reduce ExecuTorch binary size. Use when investigating binary size, running size tests, or optimizing the runtime for size-constrained deployments. - 100edge-ai
Use this skill when the user says 'edge AI', 'on-device inference', 'TensorRT', 'ONNX Runtime', 'TFLite', 'LiteRT', 'TinyML', 'quantization int8', 'Jetson', 'Coral', 'NPU', 'ExecuTorch', 'model on microcontroller'. Deploys trained models to edge hardware: export path (PyTorch/TF… - 100executorch-kb
Search the ExecuTorch tribal knowledge base covering QNN, XNNPACK, Vulkan, CoreML, Arm, and Cadence backends, quantization recipes, export pitfalls, runtime errors, and SoC compatibility. Use when debugging ExecuTorch errors, choosing quantization configs, checking backend op… - 100pytorch-model-trainer
Build pytorch model trainer operations. Auto-activating skill for ML Training. Triggers on: pytorch model trainer, pytorch model trainer Part of the ML Training skill category. Use when working with pytorch model trainer functionality. Trigger with phrases like "pytorch model… - 100mlenergy-data
Answer natural-language questions about The ML.ENERGY Benchmark (LLM and diffusion inference energy/latency/throughput measurement data on GPUs) using the `mlenergy-data` Python toolkit. Use when the user asks about energy per token, energy per image/video, throughput, ITL…