ml-inference-optimization
/ml-inference-optimization
This skill should be used when the user asks to optimize, export, serve, compress, or accelerate ML inference. PROACTIVELY activate for: (1) latency, throughput, p95/p99, batching, concurrency, KV cache, memory, or cost issues, (2) quantization INT8/INT4, GPTQ, AWQ, bitsandbytes, pruning, sparsity, distillation, (3) ONNX export, ONNX Runtime, TensorRT, TorchScript, torch.compile, XLA, OpenVINO, Core ML, TFLite, (4) Triton, TorchServe, TF Serving, BentoML, Seldon, KServe configuration, (5) edge deployment, CPU/GPU/TPU/Inferentia serving. Provides: hardware-aware inference optimization and safe
- Author
- JosiahSiegel ↗
- Repository
- claude-plugin-marketplace ↗
- Licence
- MIT
- Last repository change
- 18.06.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 6
No signals detected. Repository changed on 18.06.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
This skill should be used when the user asks to optimize, export, serve, compress, or accelerate ML inference. PROACTIVELY activate for: (1) latency, throughput, p95/p99, batching, concurrency, KV cache, memory, or cost issues, (2) quantization INT8/INT4, GPTQ, AWQ, bitsandbytes, pruning, sparsity, distillation, (3) ONNX export, ONNX Runtime, TensorRT, TorchScript, torch.compile, XLA, OpenVINO, Core ML, TFLite, (4) Triton, TorchServe, TF Serving, BentoML, Seldon, KServe configuration, (5) edge deployment, CPU/GPU/TPU/Inferentia serving. Provides: hardware-aware inference optimization and safe
Original author description
This skill should be used when the user asks to optimize, export, serve, compress, or accelerate ML inference. PROACTIVELY activate for: (1) latency, throughput, p95/p99, batching, concurrency, KV cache, memory, or cost issues, (2) quantization INT8/INT4, GPTQ, AWQ, bitsandbytes, pruning, sparsity, distillation, (3) ONNX export, ONNX Runtime, TensorRT, TorchScript, torch.compile, XLA, OpenVINO, Core ML, TFLite, (4) Triton, TorchServe, TF Serving, BentoML, Seldon, KServe configuration, (5) edge deployment, CPU/GPU/TPU/Inferentia serving. Provides: hardware-aware inference optimization and safe
How it works
- Overview
- Optimization Workflow
- Export and Compilation
- Batching and Concurrency
- Triton Inference Server Configuration
- Precision and Quantization
- Pruning, Sparsity, and Distillation
- TensorRT Compilation Recipes (trtexec)
- Serving Runtimes
- Memory Optimization
SKILL.md preview
---
name: ml-inference-optimization
description: |
This skill should be used when the user asks to optimize, export, serve, compress, or accelerate ML inference. PROACTIVELY activate for: (1) latency, throughput, p95/p99, batching, concurrency, KV cache, memory, or cost issues, (2) quantization INT8/INT4, GPTQ, AWQ, bitsandbytes, pruning, sparsity, distillation, (3) ONNX export, ONNX Runtime, TensorRT, TorchScript, torch.compile, XLA, OpenVINO, Core ML, TFLite, (4) Triton, TorchServe, TF Serving, BentoML, Seldon, KServe configuration, (5) edge deployment, CPU/GPU/TPU/Inferentia serving. Provides: hardware-aware inference optimization and safe benchmarking.
---
# ML Inference Optimization
## Overview
Use this skill for reducing inference latency, increasing throughput, shrinking memory, lowering cost, and deploying optimized models safely. Optimization must be benchmark-driven: define workload, input shapes, concurrency, SLOs, hardware, runtime, numerical tolerance, and quality metrics before changing the model.
## Optimization Workflow
1. Establish a baseline with realistic data, preprocessing, postprocessing, network overhead, warmup, and concurrency.
2. Profile bottlenecks: CPU preprocessing, model compute, memory bandwidth, GPU utilization, serialization, queueing, retrieval, or downstream calls.
3. Apply the least risky optimization first: batching, compilation, precision, runtime tuning, then compression.
4. Revalidate accuracy, calibration, fairness slices, robustness, and numerical stability after every change.
5. Measure p50, p95, p99 latency, throughput, memory, cold start, error rate, and cost per 1,000 predictions.
## Export and Compilation
Export paths should be chosen by deployment target. Compare exported output with framework output on representative inputs. Define tolerances per output type. Include preprocessing and postprocessing in the benchmark; many failures occur outside the core model.
### PyTorch to ONNX Export Blueprint
```python
import torch
import torch.nn as nn
def export_to_onnx(model: nn.Module, dummy_input: torch.Tensor, onnx_path: str):
model.eval()
# Define input and output names, and dynamic axes for batch size and sequence length
input_names = ["input_ids"]
output_names = ["logits"]
dynamic_axes = {
"input_ids": {0: "batch_size", 1: "seq_len"},
"logits": {0: "batch_size"}
}
torch.onnx.export(
model,
dummy_input,
onnx_path,
export_params=True,
opset_version=17, # Modern stable opset version
do_constant_folding=True,
input_names=input_names,
output_names=output_names,
dynamic_axes=dynamic_axes
)
```
### PyTorch `torch.compile` Optimization Setup
`torch.compile` leverages the Inductor compiler for massive speedups on modern GPUs.
```python
import torch
def compile_model(model: torch.nn.Module, mode: str = "default"):
# Modes:
# - "default": Balance compile time and inference speed
# - "reduce-overhead": Reduce framework overhead (uses CUDA graphs, good for small batch sizes)
# - "max-autotune": Perform exhaustive kernel tuning (longest compile time, fastest runtime)
optimized_model = torch.compile(
model,
mode=mode,
fullgraph=False,
dynamic=True # Handles variable sequence lengths/batch sizes efficiently
)
return optimized_model
```
## Batching and Concurrency
Dynamic batching is often the biggest serving win for GPU endpoints. Tune max batch size and queue delay together: larger batches improve throughput but can harm tail latency. For LLMs, distinguish prefill and decode phases; sequence length, KV cache, and concurrent requests dominate memory. Use continuous batching or specialized LLM serving runtimes when appropriate.
For CPU services, concurrency may improve utilization until memory bandwidth, GIL/runtime constraints, or thread oversubscription dominate. Pin thread counts for BLAS/OpenMP/ONNX Runtime and benchmark rather than accepting defaults.
## Triton Inference Server Configuration
Triton is the industry standard for high-performance multi-framework serving.
### Triton `config.pbtxt` Template
```protobuf
name: "text_classifier"
platform: "onnxruntime_onnx"
max_batch_size: 32
input [
{
name: "input_ids"
data_type: TYPE_INT64
dims: [ -1 ] # Dynamic sequence length
}
]
output [
{
name: "logits"
data_type: TYPE_FP32
dims: [ 5 ] # Number of classes
}
]
# Enable Dynamic Batching
dynamic_batching {
max_queue_delay_microseconds: 5000 # Wait up to 5ms for concurrent requests
preferred_batch_size: [ 8, 16, 32 ]
}
# GPU Instance Groups
instance_group [
{
count: 2 # Host 2 model execution instances per GPU
kind: KIND_GPU
}
]
```
## Precision and Quantization
| Method | Best for | Caveats |
|---|---|---|
| fp16/bf16 | GPU/accelerator throughput and memory reduction | Validate numerically sensitive ops |
| INT8 post-training quantization | CNNs, transformers, tabular neural nets with calibration data | Calibration set must represent production |
| Quantization-aware training | Accuracy-sensitive low precision deployments | Adds training complexity |
| INT4/GPTQ/AWQ/bitsandbytes | Large language model memory reduction | Can degrade reasoning, rare tokens, calibration, tool use |
| Weight-only quantization | LLM serving where activations remain higher precision | Speedup depends on kernels/hardware |
Quantization changes model behavior. Re-run task metrics, calibration, slice metrics, and safety checks. For RAG or embedding models, verify retrieval recall and embedding similarity distribution after quantization.
## Pruning, Sparsity, and Distillation
Pruning removes weights, channels, heads, or layers. Unstructured sparsity may not speed inference without hardware/runtime support; structured pruning is more likely to reduce latency. Knowledge distillation trains a smaller student to mimic a larger teacher.
## TensorRT Compilation Recipes (trtexec)
NVIDIA TensorRT compiled engines deliver maximum throughput on GPUs.
### 1. Build FP16 Engine with Dynamic Shapes
```bash
trtexec --onnx=model.onnx \
--saveEngine=model.engine \
--fp16 \
--minShapes=input_ids:1x1 \
--optShapes=input_ids:16x128 \
--maxShapes=input_ids:32x512
```
### 2. Build INT8 Engine with Calibration Data
```bash
trtexec --onnx=model.onnx \
--saveEngine=model_int8.engine \
--int8 \
--calib=calibration_profile.cache \
--minShapes=input_ids:1x1 \
--optShapes=input_ids:16x128 \
--maxShapes=input_ids:32x512
```
## Serving Runtimes
| Runtime | Use when |
|---|---|
| Triton Inference Server | Multi-framework GPU/CPU serving, dynamic batching, ensembles, model repository, high throughput |
| TorchServe | PyTorch model serving with handlers and management APIs |
| TensorFlow Serving | TensorFlow SavedModel serving and versioned model management |
| ONNX Runtime Server/custom API | Portable ONNX inference with lightweight service control |
| BentoML | Packaging models as APIs, deployment flexibility, Python-centric workflows |
| Seldon/KServe | Kubernetes-native model serving, canaries, inference graphs, enterprise ops |
| vLLM / TGI | Continuous batching, PagedAttention, KV cache management, tensor parallel LLM serving |
Use model repository versioning and readiness checks. Separate health checks from expensive inference. For GPU serving, avoid loading multiple memory-heavy models per device unless capacity is proven.
## Memory Optimization
For transformers, memory is driven by weights plus KV cache. Reduce memory with quantization, tensor parallelism, smaller max sequence length, paged attention, cache eviction, smaller batch/concurrency, or distilled models. For diffusion and vision models, use attention slicing, VAE tiling, fp16/bf16, memory-efficient attention, and model offload where latency allows.
## Edge Deployment
Edge constraints include binary size, RAM, thermal throttling, offline operation, intermittent updates, privacy, and hardware acceleration. Prefer TFLite/Core ML/OpenVINO/ONNX depending on target. Test on real devices.
## Benchmarking Pitfalls
- Measuring only warm single-request latency while production has concurrent traffic.
- Ignoring tokenization, image decode, retrieval, serialization, or network time.
- Reporting average latency instead of p95/p99.
- Comparing quantized and baseline models on different data.
- Building TensorRT engines for one shape then serving unsupported dynamic shapes.
- Increasing batch size until throughput improves but SLO fails.
## Sources
- NVIDIA Triton Inference Server documentation: https://docs.nvidia.com/deeplearning/triton-inference-server/
- NVIDIA TensorRT documentation: https://docs.nvidia.com/deeplearning/tensorrt/
- ONNX Runtime documentation: https://onnxruntime.ai/docs/
- PyTorch export and compilation docs: https://pytorch.org/docs/stable/
- TensorFlow Lite documentation: https://www.tensorflow.org/lite
- BentoML documentation: https://docs.bentoml.com/
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · JosiahSiegel/claude-plugin-marketplace | 100 | MIT |
| j4flmao/agent-skills | 95 | MIT |
Database snapshot: 13.11.2026 · package version c88d9f92f6e1 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90model-inference-optimize
优化实际模型推理链路,将正确性对齐、分段 profiling、显存与数据搬运、TensorRT/ONNX/PyTorch 后端、attention/kernel、FP8/compile、缓存与少步采样、质量回归、GPU 成本和服务验收串成同一实验闭环。当用户要求推理提速、降低显存或 GPU 成本、复现模型效果、定位 GPU 利用率低、优化图像/视频/扩散模型或自托管 LLM 时使用,提供 model inference performance optimization 的脱敏案例。纯 API 调用、提示词或 token 用量优化,以及电脑卡顿,不进入本技能的模型推理实验流程。 - 90ml-model-serving
Use this skill when deploying models for inference: TorchServe, BentoML, Ray Serve, KServe, Seldon Core, model deployment, A/B testing, autoscaling, canary deployment, model versioning, inference optimization. This skill enforces: framework selection based on model type,… - 90inference-serving-topology
LLM/model inference serving architecture: the engine → serving → orchestration layering (vLLM/SGLang/TensorRT-LLM, Triton, KServe/Ray Serve), KV-cache & continuous batching, prefill-decode disaggregation, and scaling. Architect-level topology, not model training. USE WHEN:… - 100edge-ai
Use this skill when the user says 'edge AI', 'on-device inference', 'TensorRT', 'ONNX Runtime', 'TFLite', 'LiteRT', 'TinyML', 'quantization int8', 'Jetson', 'Coral', 'NPU', 'ExecuTorch', 'model on microcontroller'. Deploys trained models to edge hardware: export path (PyTorch/TF… - 100tei-triton-serving
High-performance serving of embedding and reranker models in production. Covers HuggingFace Text Embeddings Inference (TEI) on GPU, Docker deployment, ONNX/FP16 quantization, dynamic batching, NVIDIA Triton for co-serving embedding + reranker + LLM, and NVIDIA NIM as a managed… - 100fai-inference-optimization
Optimize model inference latency and throughput — batching, quantization, caching, KV-cache reuse, and hardware/SKU selection. Use this skill when responses are too slow, cost per token is too high, or scaling under load — even if the user only says "make the model faster" or "reduce latency".