quantization-deployment
/quantization-deployment
Kwantyzuje wytrenowany model do INT8 lub INT4 dla wnioskowania, kalibruje zakresy i warunkuje wydanie zmierzonym spadkiem jakości.
- Autor
- Amey-Thakur ↗
- Repozytorium
- AI-SKILLS ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 05.09.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 05.09.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Kwantyzuje wytrenowany model do INT8 lub INT4 dla wnioskowania, kalibruje zakresy i warunkuje wydanie zmierzonym spadkiem jakości.
Oryginalny opis (EN)
Quantize a trained model to INT8 or INT4 for inference, calibrate the ranges, and gate the release on a measured quality regression. Use when serving needs lower latency and memory and you will spend effort keeping accuracy inside a defined budget.
Jak działa
- Method
- Checks
- Boundaries
Podgląd SKILL.md
--- name: quantization-deployment description: Quantize a trained model to INT8 or INT4 for inference, calibrate the ranges, and gate the release on a measured quality regression. Use when serving needs lower latency and memory and you will spend effort keeping accuracy inside a defined budget. --- # Quantization deployment Quantization maps floating-point weights and activations to low-bit integers, cutting memory 2x to 4x and letting integer tensor cores do the math faster. The catch is that it is lossy: pick the wrong ranges or scheme and accuracy falls off a cliff, often on exactly the inputs you never tested. The discipline is to calibrate ranges on real data, choose the scheme per tensor, and refuse to ship until a quality gate passes. ## Method 1. **Start with post-training quantization, escalate to QAT only on failure.** Post-training quantization (PTQ) needs no retraining: convert an existing FP model and calibrate. If PTQ misses the budget, quantization-aware training (QAT) simulates the rounding during fine-tuning and usually recovers most of the gap, at the cost of a training run. Do the cheap thing first. 2. **Calibrate activation ranges on representative data.** Weights have fixed values, but activation ranges depend on inputs. Push a few hundred representative samples through the model to record per-tensor histograms, then set clipping thresholds. Entropy or percentile calibration (clipping the top 0.01 percent of outliers) beats plain min/max, which a single outlier ruins. 3. **Tame activation outliers before they wreck the scale.** In transformers a few channels carry huge activation magnitudes that stretch the range and crush everything else. SmoothQuant shifts that difficulty from activations into weights so a per-tensor activation scale still fits. Apply it before choosing thresholds, not after. 4. **Match the scheme to the tensor.** Use per-channel (per-output-channel) scales for weights and per-tensor scales for activations; per-channel weight quantization alone recovers much of the loss. Prefer symmetric quantization for weights and asymmetric with a nonzero zero-point for one-sided activations like post-ReLU outputs. 5. **Choose bit width against the model's tolerance.** INT8 is the safe default and often loses under 1 percent accuracy. INT4 doubles the saving again but needs group-wise scales (for example groups of 128 weights) and a salient-weight scheme like GPTQ or AWQ to stay usable. Do not reach for INT4 before INT8 is validated. 6. **Keep sensitive layers in higher precision.** The first and last layers, and any layer with wide activation ranges, often dominate the error. Run a sensitivity sweep that quantizes one layer at a time and measures the drop, then leave the worst offenders in INT8 or FP16. Mixed precision trades a little speed for a lot of accuracy. 7. **Gate the release on a measured regression.** Fix the budget before you quantize (top-1 within 1 percent, or perplexity increase under 2 percent) and evaluate on a held-out set separate from calibration data. Block the deploy if the gate fails, and record before/after numbers and the scheme in the release notes. ## Checks - Does the quantized model meet the pre-stated budget on a held-out set separate from calibration data? - Were activation ranges calibrated on representative inputs with outlier clipping, not raw min/max? - Are weights per-channel and activations per-tensor, with symmetric or asymmetric chosen to fit each? - Did a per-layer sensitivity sweep justify which layers stay higher precision? ## Boundaries This targets integer quantization for inference serving. Floating-point half formats (FP16, BF16, FP8) and their loss scaling are mixed-precision-deployment. Making the resulting INT8 matmuls actually hit the integer tensor cores is tensor-core-utilization. The accuracy budget itself is a product decision: quantization measures the cost, the owner decides if it is acceptable.
Wersje
W bazie nie ma innych skilli o nazwie quantization-deployment.
Migawka bazy z 13.11.2026 · wersja paczki d36fadf82563 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100model-quantization-tool
Prawdopodobnie pomaga w kwantyzacji modeli uczenia maszynowego przed wdrożeniem, w ramach kategorii ML Deployment. - 100ptq
Kwantyzuje model po treningu (NVFP4, FP8, INT8, INT4 AWQ) w ModelOpt i tworzy checkpoint dla Hugging Face lub TensorRT-LLM. - 100modelopt-ptq
Kwantyzuje model po treningu w ModelOpt do checkpointu HuggingFace: NVFP4, FP8, INT8 lub INT4 AWQ, także MoE i VLM. - 100ptq
Kwantyzuje model po treningu (PTQ) przez ModelOpt do formatów NVFP4, FP8, INT8 i INT4 AWQ, tworząc skwantyzowany checkpoint HuggingFace z wytrenowanego modelu, także MoE i VLM. - 95awq-quantization
Kwantyzuje duże modele językowe do 4 bitów metodą AWQ, z mniejszym spadkiem dokładności i szybszym wnioskowaniem niż GPTQ. - 100cortex-model
Buduje potok ML od danych przez wytrenowany model po endpoint serwujący, na przykład dla klasyfikacji, regresji i potoków predykcyjnych.