Przejdź do treści
Biblioteka skilli
AI i agenci · Automatyzacja

deployment

/deployment

Udostępnia skwantyzowany lub nieskwantyzowany checkpoint LLM jako endpoint API zgodny z OpenAI przez vLLM, SGLang lub TRT-LLM oraz mierzy przepustowość.

Licencja
MIT
Ostatnia zmiana repozytorium
27.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
10
Skrypty
1
Komentarze HTML
0
Adresy URL w SKILL.md
6
Przestarzałość

Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 1 skrypt w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/deploy.sh

Co robi i kiedy

Udostępnia skwantyzowany lub nieskwantyzowany checkpoint LLM jako endpoint API zgodny z OpenAI przez vLLM, SGLang lub TRT-LLM oraz mierzy przepustowość.

Wyzwalacze z opisu

deploy modelserve modelstart vLLM serverlaunch SGLangTRT-LLM deployAutoDeploybenchmark throughputserve checkpoint
Oryginalny opis (EN)

Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).

Jak działa

  1. Quick Start
  2. Decision Flow
  3. Error Handling
  4. Unsupported Models
  5. Success Criteria
Sekcje z nagłówków SKILL.md, w oryginale · 272 linii, 1 650 słów
Podgląd SKILL.md
---
name: deployment
description: Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).
license: Apache-2.0
---

# Deployment Skill

Serve a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).

## Quick Start

Prefer `$SKILL_DIR/scripts/deploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.

```bash
# Start vLLM server with a ModelOpt checkpoint
"$SKILL_DIR/scripts/deploy.sh" start --model ./qwen3-0.6b-fp8

# Start with SGLang and tensor parallelism
"$SKILL_DIR/scripts/deploy.sh" start --model ./llama-70b-nvfp4 --framework sglang --tp 4

# Start from HuggingFace hub
"$SKILL_DIR/scripts/deploy.sh" start --model nvidia/Llama-3.1-8B-Instruct-FP8

# Test the API
"$SKILL_DIR/scripts/deploy.sh" test

# Check status
"$SKILL_DIR/scripts/deploy.sh" status

# Stop
"$SKILL_DIR/scripts/deploy.sh" stop
```

The script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start/stop/restart/status), health check polling, and API testing.

## Decision Flow

### 0. Check workspace (multi-user / Slack bot)

If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:

```bash
ls "$MODELOPT_WORKSPACE_ROOT/<session_id>/" 2>/dev/null
```

If the user says "deploy the model I just quantized" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.

### 1. Identify the checkpoint

Determine what the user wants to deploy:

- **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output/`, `outputs/`, `exported_model/`, or the `--export_path` used in the PTQ command.
- **HuggingFace model hub** (e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`): use directly
- **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill

> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references/trtllm.md`.

Check the quantization format if applicable:

```bash
cat <checkpoint_path>/hf_quant_config.json 2>/dev/null || echo "No hf_quant_config.json"
```

If not found, also check `config.json` for a `quantization_config` section with `quant_method: "modelopt"`. If neither exists, the checkpoint is unquantized.

### 2. Choose the framework

If the user hasn't specified a framework, recommend based on this priority:

| Situation | Recommended | Why |
|-----------|-------------|-----|
| General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible |
| Best SGLang model support | **SGLang** | Strong DeepSeek/Llama 4 support |
| Maximum optimization | **TRT-LLM** | Best throughput via engine compilation |
| Mixed-precision / AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |

Check the support matrix in `references/support-matrix.md` to confirm the model + format + framework combination is supported.

### 3. Check the environment

Use the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM/Docker/bare metal detection. After completing it you should know: GPU model/count, local or remote, and execution environment.

Then check the **deployment framework** is installed:

```bash
python -c "import vllm; print(f'vLLM {vllm.__version__}')" 2>/dev/null || echo "vLLM not installed"
python -c "import sglang; print(f'SGLang {sglang.__version__}')" 2>/dev/null || echo "SGLang not installed"
python -c "import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')" 2>/dev/null || echo "TRT-LLM not installed"
```

If not installed, consult `references/setup.md`.

**GPU memory estimate** (to determine tensor parallelism):

- BF16: `params × 2 bytes` (8B ≈ 16 GB)
- FP8: `params × 1 byte` (8B ≈ 8 GB)
- FP4: `params × 0.5 bytes` (8B ≈ 4 GB)
- Add ~2-4 GB for KV cache and framework overhead

If the model exceeds single GPU memory, use tensor parallelism (`-tp <num_gpus>`).

### 4. Deploy

Read the framework-specific reference for detailed instructions:

| Framework | Reference file |
|-----------|---------------|
| vLLM | `references/vllm.md` |
| SGLang | `references/sglang.md` |
| TRT-LLM | `references/trtllm.md` |

**Quick-start commands** (for common cases):

#### vLLM

```bash
# Serve as OpenAI-compatible endpoint
python -m vllm.entrypoints.openai.api_server \
    --model <checkpoint_path> \
    --quantization modelopt \
    --tensor-parallel-size <num_gpus> \
    --host 0.0.0.0 --port 8000
```

For NVFP4 checkpoints, use `--quantization modelopt_fp4`.

> **NVFP4 on Blackwell B300/GB300 (sm_103) needs a CUDA-13 image.** From v0.20.0
> on, release tags are CUDA-13 unsuffixed (e.g. `vllm/vllm-openai:v0.26.0`) with
> `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round
> (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust
> the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob
> of your platform's child manifest (arm64 Grace/GB300, amd64 x86);
> `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103
> FP4 kernel**, so vLLM loads the checkpoint then dies at engine init with `CUDA
> error: no kernel image is available for execution on the device` (affects the
> `flashinfer` and `cutlass` NVFP4 backends; `marlin` separately fails on
> non-64-divisible layer dims).
> Cross-check via
> `recipes.vllm.ai/<org>/<model>?hardware=b300` (JS-rendered — fetch the raw
> markdown at `github.com/vllm-project/recipes/blob/main/<org>/<model>.md`). For
> multimodal models on sm_103, also pass `--mm-encoder-attn-backend TRITON_ATTN`
> (the default CuTe ViT flash-attn asserts "Only SM 10.x and 11.x").

#### SGLang

```bash
python -m sglang.launch_server \
    --model-path <checkpoint_path> \
    --quantization modelopt \
    --tp <num_gpus> \
    --host 0.0.0.0 --port 8000
```

For NVFP4 checkpoints, use `--quantization modelopt_fp4`.

> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog
> of `recipes.vllm.ai`): `docs.sglang.io/cookbook/<category>/<org>/<model>` (e.g.
> `.../autoregressive/DeepSeek/DeepSeek-V4`) — authoritative for parallelism, MoE
> backends, strategy flags, Docker image, and min version. Select the variant via
> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The
> page is **JS-rendered** — fetch the raw markdown at
> `raw.githubusercontent.com/sgl-project/sglang/main/docs_new/cookbook/<category>/<org>/<model>.mdx`.
> SM120 (RTX PRO 6000) needs the `lmsysorg/sglang:dev` nightly (`:latest` lacks
> SM120). See `references/sglang.md` for the full backend/flag matrix.

#### TRT-LLM (direct)

```python
from tensorrt_llm import LLM, SamplingParams
llm = LLM(model="<checkpoint_path>")
outputs = llm.generate(["Hello, my name is"], SamplingParams(temperature=0.8, top_p=0.95))
```

#### TRT-LLM AutoDeploy

For AutoQuant or mixed-precision checkpoints, see `references/trtllm.md`.

### 5. Verify the deployment

After the server starts, verify it's healthy:

```bash
# Health check
curl -s http://localhost:8000/health

# List models
curl -s http://localhost:8000/v1/models | python -m json.tool

# Test generation
curl -s http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "<model_name>",
        "prompt": "The capital of France is",
        "max_tokens": 32
    }' | python -m json.tool
```

All checks must pass before reporting success to the user.

### 5b. Benchmark throughput/latency (optional)

If the user asks to benchmark, measure throughput/latency, or compare precisions,
use **AIPerf** (Apache-2.0, OpenAI-compatible client benchmark). See
`references/benchmarking.md` for install, the pre-benchmark coherence gate, the
`aiperf profile` flags (notably `--extra-inputs ignore_eos:true`), suggested
token shapes, and how to read `profile_export_aiperf.json`.

### 6. Remote deployment (SSH/SLURM)

If a cluster config exists (`~/.config/modelopt/clusters.yaml`, `.agents/clusters.yaml`, or `.claude/clusters.yaml`), or the user mentions running on a remote machine:

0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's `slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**

1. **Source remote utilities:** Load the common skill, then resolve
   `remote_exec.sh` from that skill's root.

   ```bash
   source "<common-skill-dir>/remote_exec.sh"
   remote_load_cluster
   remote_check_ssh
   remote_detect_env
   ```

2. **Sync the checkpoint** (only if it was produced locally):

   If the checkpoint path is a remote/absolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with `remote_run "ls <checkpoint_path>/config.json"`. Only sync if the checkpoint is local:

   ```bash
   remote_sync_to <local_checkpoint_path> <session_id>/<model>/checkpoints/
   ```

3. **Deploy based on remote environment:**

   - **SLURM** — see the common skill's `slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). After submitting, register the job and set up monitoring per the **monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.

   - **Bare metal / Docker** — use `remote_run` to start the server directly:

     ```bash
     remote_run "nohup python -m vllm.entrypoints.openai.api_server --model <path> --port 8000 > deploy.log 2>&1 &"
     ```

4. **Verify remotely:**

   ```bash
   remote_run "curl -s http://localhost:8000/health"
   remote_run "curl -s http://localhost:8000/v1/models"
   ```

5. **Report the endpoint** — include the remote hostname and port so the user can connect (e.g., `http://<node_hostname>:8000`). For SLURM, note that the port is only reachable from within the cluster network.

For NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.

## Error Handling

| Error | Cause | Fix |
|-------|-------|-----|
| `CUDA error: an illegal memory access` on an **NVFP4 MoE** (`trtllm_fused_moe_dev_kernel.cu`, `deepgemm`) | Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500 | Try `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput`. **Not always a fix** — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas. |
| `CUDA out of memory` | Model too large for GPU(s) | Increase `--tensor-parallel-size` or use a smaller model |
| `quantization="modelopt" not recognized` | vLLM/SGLang version too old | Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10 |
| `hf_quant_config.json not found` | Not a ModelOpt-exported checkpoint | Re-export with `export_hf_checkpoint()`, or remove `--quantization` flag |
| `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |
| `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references/support-matrix.md` |

## Unsupported Models

If the model is not in the validated support matrix (`references/support-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized/unquantized layer confusion. Read `references/unsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.

## Success Criteria

1. Server process is running and healthy (`/health` returns 200)
2. Model is listed at `/v1/models`
3. Test generation produces coherent output
4. Server URL and port are reported to the user
5. If benchmarking was requested, throughput/latency numbers are reported
Pliki w paczce (10)
  • SKILL.md 13 KB
  • references/benchmarking.md 6 KB
  • references/setup.md 2 KB
  • references/sglang.md 5 KB
  • references/support-matrix.md 3 KB
  • references/trtllm.md 3 KB
  • references/unsupported-models.md 5 KB
  • references/vllm.md 3 KB
  • scripts/deploy.sh 19 KB
  • tests/evals.json 3 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · nvidia/model-optimizer100MIT
ericrisco/rsc-harness100MIT
atretyak1985/swarmery100Apache-2.0
axone-protocol/contracts100BSD-3-Clause
antoinebou12/uml-mcp100MIT
CrowdStrike/fusion-skills100MIT
iWiiCK/iwiick-ai-agent-plugin100MIT
andreibesleaga/gabbe100CC-BY-4.0
digitalocean-labs/do-app-platform-skills100MIT

Migawka bazy z 13.11.2026 · wersja paczki 7bd0bf868a4a (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100modelopt-deployment
    Uruchamia skwantyzowany lub zwykły model LLM jako endpoint API zgodny z OpenAI na vLLM, SGLang lub TRT-LLM.
  • 100minicpm5-deploy-vllm
    Udostępnia MiniCPM5-1B lub MiniCPM5-2B przez vLLM jako serwer HTTP zgodny z OpenAI, do produkcyjnej obsługi o dużej przepustowości na GPU NVIDIA.
  • 100debug-inference
    Diagnozuje klientów inferencji korzystających z dołączonego dostawcy i jego natywnego endpointu, w tym hostowane API oraz lokalne Ollama, vLLM, SGLang, TRT-LLM, LM Studio i NIM.
  • 100minicpm5-deploy-sglang
    Udostępnia MiniCPM5-1B lub MiniCPM5-2B przez SGLang jako serwer HTTP zgodny z OpenAI, z pamięcią podręczną prefiksów RadixAttention i wbudowanym parserem wywołań narzędzi, na GPU NVIDIA.
  • 100vllm
    Podaje referencję API Pythona vLLM v0.19.0: generowanie wsadowe, czat, embeddingi, wyjścia strukturalne, LoRA i serwer zgodny z OpenAI.
  • 100ptq
    Kwantyzuje model po treningu (PTQ) przez ModelOpt do formatów NVFP4, FP8, INT8 i INT4 AWQ, tworząc skwantyzowany checkpoint HuggingFace z wytrenowanego modelu, także MoE i VLM.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.