Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

ml-inference-optimization

/ml-inference-optimization

Diagnozuje i przyspiesza wnioskowanie modeli językowych: opóźnienie, przepustowość, pamięć, silnik, batching, kwantyzację i decodowanie spekulacyjne.

Repozytorium
agent-skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
10.11.2026

Ocena i bezpieczeństwo

95/ 100Klasa: dobry
Format20 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 10.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Diagnozuje i przyspiesza wnioskowanie modeli językowych: opóźnienie, przepustowość, pamięć, silnik, batching, kwantyzację i decodowanie spekulacyjne.

Wyzwalacze z opisu

llm inference optimizationreduce latency llmtime to first tokenttfttokens per secondllm throughputvllm vs tgispeculative decoding
Oryginalny opis (EN)

Use this skill when the user says 'llm inference optimization', 'reduce latency llm', 'time to first token', 'ttft', 'tokens per second', 'llm throughput', 'vllm vs tgi', 'speculative decoding', 'continuous batching', 'serve llm cheaper'. Diagnoses and optimizes LLM inference end to end: latency/throughput metrics, memory budget, engine choice, batching, quantization, KV cache, parallelism and speculative decoding. Do NOT use for: deep KV-cache paging internals (paged-attention), quantization algorithm details (extreme-quantization-mastery), TensorRT engine builds (tensorrt-optimization), clas

Jak działa

  1. Purpose
  2. Agent Protocol
  3. Metrics
  4. Memory Budget
  5. Optimization Levers (roughly in order)
  6. Example: vLLM Offline Batch + Server
  7. Benchmarking
  8. Decision Rules
  9. Anti-Patterns
  10. References
Sekcje z nagłówków SKILL.md, w oryginale · 102 linii, 787 słów
Podgląd SKILL.md
---
name: ml-inference-optimization
description: >
  Use this skill when the user says 'llm inference optimization', 'reduce latency llm', 'time to first token', 'ttft', 'tokens per second', 'llm throughput', 'vllm vs tgi', 'speculative decoding', 'continuous batching', 'serve llm cheaper'. Diagnoses and optimizes LLM inference end to end: latency/throughput metrics, memory budget, engine choice, batching, quantization, KV cache, parallelism and speculative decoding. Do NOT use for: deep KV-cache paging internals (paged-attention), quantization algorithm details (extreme-quantization-mastery), TensorRT engine builds (tensorrt-optimization), classical model serving infra (ml-model-serving).
version: "1.0.0"
author: "j4flmao"
license: "MIT"
tags: [ml, llm, inference, vllm, latency, throughput, optimization]
---

# LLM Inference Optimization

## Purpose
Serve LLMs at the required latency for the lowest cost. Decoding one token at a time is usually memory-bandwidth bound (every step reads all weights plus the KV cache), while prompt prefill is compute bound. Optimizations therefore aim to batch more requests per weight read, shrink bytes moved (quantization, KV compression), and generate more tokens per step (speculative decoding).

## Agent Protocol

### Trigger
"llm inference", "optimize inference", "ttft", "time to first token", "tokens per second", "throughput", "inter-token latency", "vllm", "sglang", "tgi", "tensorrt-llm", "speculative decoding", "serving cost".

### Input Context
- Model, GPU type/count, current engine and config.
- Traffic: prompt and output length distributions, concurrency, latency SLOs (TTFT, inter-token latency).
- Current metrics and cost per million tokens.

### Output Artifact
Bottleneck diagnosis + prioritized changes + benchmark plan. No file unless requested.

### Response Format
```
Workload: prompt {p50/p95 tokens}, output {p50/p95}, concurrency {n}
Bound: {prefill compute|decode bandwidth|KV capacity|scheduler/host}
Changes (ordered): 1) {...} 2) {...}
Benchmark: {tool + metric targets}
```

### Completion Criteria
- [ ] Baseline measured with realistic prompt/output lengths and concurrency.
- [ ] Each change measured separately for TTFT, inter-token latency, throughput, quality.
- [ ] Quality evaluated after quantization or speculative decoding config changes.

## Metrics
| Metric | Meaning | Dominated by |
|--------|---------|--------------|
| TTFT | Time to first token | Queueing + prefill (prompt length) |
| ITL / TPOT | Time between output tokens | Decode step time (weights + KV bytes / bandwidth) |
| Throughput | Output tokens/s across all requests | Batch size the KV memory allows |
| Goodput | Requests/s meeting the SLO | Trade-off between batch size and latency |

## Memory Budget
- Weights: params x bytes per param (70B BF16 ≈ 140 GB → at least 2 x 80 GB GPUs; FP8 ≈ 70 GB; 4-bit ≈ 35-40 GB).
- KV cache per token: 2 x layers x kv_heads x head_dim x bytes. Grouped-query attention models have far fewer KV heads than query heads.
- Whatever remains after weights and activations is KV capacity, which caps concurrency.

## Optimization Levers (roughly in order)
| Lever | Helps | Notes |
|-------|-------|-------|
| Dedicated engine (vLLM, SGLang, TensorRT-LLM, TGI) | Large throughput gains over naive `generate()` loops | Continuous batching, paged KV cache, fused kernels |
| Continuous batching | Throughput | New requests join at every step |
| Paged KV cache + prefix caching | Concurrency, TTFT for shared prompts | Keep shared system prompt/document first |
| FP8 weights/activations (Hopper+) | Throughput, memory | Near-lossless for most models; validate |
| 4-bit weight-only (AWQ/GPTQ) | Memory, small-batch latency | Can be slower at large batch |
| FP8 KV cache | Concurrency, long context | Validate long-context quality |
| Tensor parallelism | Latency, fit large models | Needs fast intra-node links (NVLink) |
| Speculative decoding (draft model, n-gram, EAGLE, MTP heads) | Inter-token latency at low/medium load | Gains shrink when GPU is already saturated by batching |
| Chunked prefill | Tail ITL during long prompts | Interleaves prefill chunks with decode |
| Prefill/decode disaggregation | SLOs at scale | Separate GPU pools; KV transfer between them |

## Example: vLLM Offline Batch + Server
```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen2.5-7B-Instruct",
    gpu_memory_utilization=0.90,
    max_model_len=8192,
    enable_prefix_caching=True,
)
params = SamplingParams(temperature=0.2, max_tokens=256)
outputs = llm.generate(["Summarize: ...document text...", "Translate to French: hello"], params)
for o in outputs:
    print(o.outputs[0].text)
```
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192 --enable-prefix-caching
```
Pass prompts as a list so the engine batches them; looping one prompt at a time throws away batching.

## Benchmarking
- Use the engine's benchmark tooling (e.g. `vllm bench serve`) or a load generator that replays real length distributions at fixed request rates.
- Sweep request rate; plot TTFT/ITL p50/p99 vs throughput to find the knee.
- Warm up first; report GPU type, engine version, precision, and max concurrency.

## Decision Rules
```
TTFT high, ITL fine → prefill bound or queueing: prefix caching, chunked prefill, more replicas
ITL high at low load → bandwidth bound: quantize weights, speculative decoding, TP
Throughput capped with KV usage ~100% → FP8 KV, shorter max_model_len, quantized weights, more GPUs
GPU util low with high load → host/scheduler bound: check tokenizer, detokenization, Python overhead, engine version
```

## Anti-Patterns
- Serving production traffic with `transformers` `pipeline()`/`generate()` per request.
- Benchmarking with one prompt length and concurrency 1, then sizing a fleet.
- Enabling every optimization at once without per-change measurement.
- Claiming quality is unchanged after quantization without task evals.

## References
- Related skills: `../../hardware-acceleration/paged-attention/SKILL.md`, `../../hardware-acceleration/extreme-quantization/SKILL.md`, `../../hardware-acceleration/flash-attention/SKILL.md`, `../model-serving/SKILL.md`.
- Kwon et al., PagedAttention/vLLM (arXiv:2309.06180); Leviathan et al., speculative decoding (arXiv:2211.17192).

Wersje

RepozytoriumWynikLicencja
Ta strona · j4flmao/agent-skills95MIT
JosiahSiegel/claude-plugin-marketplace100MIT

Migawka bazy z 13.11.2026 · wersja paczki f84ea4fd704b (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 95extreme-quantization-mastery
    Dobiera i stosuje kwantyzację wag, aktywacji i pamięci KV modeli LLM na sprzęcie NVIDIA oraz sprawdza utratę dokładności.
  • 100paged-attention
    Wyjaśnia zarządzanie pamięcią KV cache w PagedAttention i stroi silniki serwujące LLM, np. vLLM, pod kątem pojemności i przepustowości.
  • 100fai-inference-optimization
    Optymalizuje opóźnienia i przepustowość inferencji modeli: batching, kwantyzacja, cache, ponowne użycie KV-cache oraz dobór sprzętu i SKU.
  • 95tensorrt-llm
    Optymalizuje wnioskowanie LLM na kartach NVIDIA przez TensorRT: wyższą przepustowość, niższe opóźnienia, kwantyzację FP8 i INT4.
  • 100vllm
    Obsługuje, konfiguruje, testuje wydajność i diagnozuje serwery inferencji vLLM: Docker i Kubernetes, konfiguracja modeli z kwantyzacją, API zgodne z OpenAI, benchmarki, continuous batching i aktualizacje z rollbackiem.
  • 90inference-serving-topology
    Opisuje architekturę serwowania wnioskowania modeli i LLM: warstwy silnik, serwowanie i orkiestracja (vLLM, SGLang, TensorRT-LLM, Triton, KServe, Ray Serve), KV cache, continuous batching i skalowanie.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.