vllm-setup
/vllm-setup
Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.
- Author
- nvidia ↗
- Repository
- dgx-spark-playbooks ↗
- Licence
- Apache-2.0
- Last repository change
- 23.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 1
No signals detected. Repository changed on 23.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.
Original author description
Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.
How it works
- Steps
- Tuning parameters
SKILL.md preview
---
name: vllm-setup
description: Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.
metadata:
publisher: nvidia
hardware: DGX Station GB300
---
# vLLM Setup on DGX Station
Deploy a vLLM inference server on DGX Station with validated configuration.
## Steps
1. **Find the GB300 GPU index.** Run:
```bash
nvidia-smi --query-gpu=index,name --format=csv,noheader
```
Identify the device index for the GB300 (typically device 1). Use this index for `--gpus` below. Do NOT use `--gpus all` — mixed coherency will cause CUDA failures.
2. **Ask the user which model to serve.** If they don't have a preference, suggest:
- `nvidia/Qwen3-235B-A22B-NVFP4` — large MoE model, fits in 279 GB HBM
- `meta-llama/Llama-3.1-70B-Instruct` — solid general-purpose model
- `Qwen/Qwen3-8B` — small model for testing
3. **Check if the user has an HF_TOKEN.** Many models require HuggingFace authentication. The token must be passed inline with `-e HF_TOKEN="..."` — do not rely on shell export in background Docker tasks.
4. **Deploy the container.** Use this validated configuration:
```bash
docker pull nvcr.io/nvidia/vllm:26.01-py3
docker run -d \
--name vllm-server \
--gpus '"device=<GB300_INDEX>"' \
--ipc host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
-p 8000:8000 \
-e HF_TOKEN="<TOKEN>" \
-v "$HOME/.cache/huggingface/hub:/root/.cache/huggingface/hub" \
nvcr.io/nvidia/vllm:26.01-py3 \
vllm serve "<MODEL>" \
--max-model-len 32768 \
--gpu-memory-utilization 0.9
```
**Container version:** Use `nvcr.io/nvidia/vllm:26.01-py3`. Do NOT use 25.10 — it has a FlashInfer buffer overflow on DGX Station.
5. **Wait for the server to be ready.** Monitor logs:
```bash
docker logs -f vllm-server
```
Wait for the line indicating the server is listening on port 8000.
6. **Test the server:**
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<MODEL>",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 64
}'
```
7. **Report the result** to the user, including:
- Model loaded and serving on port 8000
- GPU memory utilization
- How to stop: `docker stop vllm-server && docker rm vllm-server`
## Tuning parameters
Adjust these based on the user's workload:
| Parameter | Default | Agent workloads | Throughput workloads |
|-----------|---------|-----------------|---------------------|
| `--max-model-len` | 32768 | 32768-65536 | 8192-16384 |
| `--gpu-memory-utilization` | 0.9 | 0.85-0.90 | 0.90-0.92 |
| `--enable-prefix-caching` | off | Enable (multi-turn reuse) | Enable |
| `--max-num-seqs` | default | 4-16 (lower latency) | 32+ (higher throughput) |
Versions
The database has no other skills named vllm-setup.
Database snapshot: 13.11.2026 · package version ca970169dd41 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100sglang-setup
Deploy an SGLang inference server on an NVIDIA DGX Station GB300 with the cu130 container, RadixAttention prefix caching, and structured JSON output support. Use when the user asks to serve a model with SGLang, start an SGLang endpoint, or needs structured-output inference on DGX Station. - 100dgx-station-inference
Resolve, tune, preflight, launch, verify, inspect, and stop exact-model inference on NVIDIA DGX Station through dgx-assist. Use for vLLM or SGLang container selection, NGC versus upstream, GPU memory utilization, CPU or KV offload, HBM fit, KV-cache sizing, ISL or context… - 100minicpm5-deploy-vllm
Serve MiniCPM5-1B or MiniCPM5-2B via vLLM as an OpenAI-compatible HTTP server. Use when the user wants high-throughput production serving on NVIDIA GPU, asks for "vLLM", "OpenAI server", "REST API for MiniCPM5", or "production deployment". - 100vllm
Complete vLLM v0.19.0 Python API reference for high-throughput LLM inference: offline batch generation, chat, embeddings, classification, structured outputs, LoRA adapters, multimodal inputs, and OpenAI-compatible server. Triggers on: vLLM questions, Python LLM serving, GPU… - 100inference-server
Start and test the prime-rl inference server. Use when asked to run inference, start vLLM, test a model, or launch the inference server. - 100spark-environment-setup
Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.