inference-builder
/inference-builder
Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.
- Author
- nvidia ↗
- Repository
- deepstream ↗
- Licence
- Apache-2.0
- Last repository change
- 22.10.2026
Assessment and security
- Files in this package
- 4
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 22.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.
Original author description
Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.
How it works
- Setup
- Key Directories
- Instructions
- Non-Obvious Knowledge
- Examples
SKILL.md preview
---
name: inference-builder
description: |
Generate deployable Vision AI pipelines with high-performance video and
streaming capabilities on NVIDIA GPUs. Activate when users want to: create
GPU-accelerated inference microservices or standalone apps for vision,
video, or streaming workloads; write or edit pipeline YAML configs; build
Docker images for GPU inference; work with models from NGC or HuggingFace;
or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.
owner: "chunlinl@nvidia.com"
metadata:
author: "Chunlin Li <chunlinl@nvidia.com>"
version: "1.0.0"
tags:
- inference
- nvidia
- deepstream
- vllm
- tensorrt-llm
- triton
- docker
- gpu
service: "inference"
reviewed: "2026-04-09"
---
# NVIDIA Inference Builder
A code generator for Vision AI pipelines with high-performance video and
streaming capabilities on NVIDIA GPUs. It turns a YAML config (+ optional
OpenAPI spec and custom processors) into a deployable inference pipeline,
packaged as a Docker container or standalone application.
## Setup
Inference Builder is part of the DeepStream mono-repo. Resolve it from the
repository root so the skill works from any subdirectory:
```bash
DEEPSTREAM_ROOT="$(git rev-parse --show-toplevel)"
INFERENCE_BUILDER_ROOT="$DEEPSTREAM_ROOT/tools/inference_builder"
source "$INFERENCE_BUILDER_ROOT/.venv/bin/activate"
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
```
### First-time venv setup
```bash
cd "$INFERENCE_BUILDER_ROOT"
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
### Getting current CLI usage
Always run with `--help` for authoritative flags — do NOT rely on memorized
syntax:
```bash
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
```
All paths below are relative to `tools/inference_builder/` unless stated otherwise.
## Key Directories
| Path | What lives there |
|------|-----------------|
| `builder/samples/` | Working examples for every backend — read `builder/samples/README.md` for the full index, then start from the closest matching example |
| `schemas/config.schema.json` | Main config schema — all fields, data types, and validation rules |
| `schemas/README.md` | Comprehensive reference: backends, parameters, data types, preprocessors, responders, routing |
| `schemas/backends/` | Per-backend schemas and parameter definitions |
| `templates/backend/` | Jinja templates for each inference backend |
| `templates/responder/` | Jinja templates for server endpoint handlers |
| `lib/` | Shared runtime libraries included in generated pipelines |
| `builder/tests/` | Integration test framework (see `run_tests.sh --help`) |
## Instructions
1. **Browse samples & schemas** — find the closest match in `builder/samples/`;
consult `schemas/config.schema.json` and `schemas/backends/` for field reference
2. **Write YAML config** — validate against the schema; consult
`schemas/README.md` for data types, preprocessors, responders, and routing.
Prefer choices that keep the hot path on GPU and skip avoidable GPU↔CPU
round-trips; prefer GPU acceleration across the whole pipeline — decode/
encode, scale & color convert, batch, preprocess, infer, track, and overlay
3. **Generate pipeline** — `python builder/main.py <config.yaml> [options]`
4. **Prepare model repository** — download models from NGC/HuggingFace into
the model_repo directory
5. **Generate nvinfer config** (DeepStream only) — check if the downloaded
model already includes runtime config files (nvdsinfer_config.yaml,
nvdspreprocess_config.yaml); if not, generate them and place alongside
the model. Skip this step entirely for non-DeepStream backends
6. **Select platform & build Docker image** — read `doc/platform-guide.md` to
choose the correct base image and Dockerfile for the target hardware
(x86_64, Jetson/Tegra, DGX Spark); then build the image
7. **Run & test** — run the container with the model repository mounted;
for serverless pipelines the container runs to completion; for HTTP
servers (fastapi/triton/nim) send test requests to verify. If GPU
resources are available, perform an end-to-end smoke test against the
running container to validate the full pipeline on real hardware
## Non-Obvious Knowledge
- If the user's model request is ambiguous (e.g., unclear model variant,
version, precision, or source), always ask for clarification before
proceeding — do not guess.
- Before using any model, gather as much information as possible about it:
version, architecture, input/output shapes, precision, and expected
preprocessing. Cross-check this information against the pipeline YAML
config and any runtime config (e.g., nvdsinfer_config.yaml) to ensure
consistency — mismatches in input dims, scale factors, or label files
are common sources of silent failures.
- Config YAML supports runtime env-var substitution: `$ENV_VAR|default_value`
(auto-typed based on the default).
- Server type `serverless` does NOT need an OpenAPI spec or server definition.
- Custom processor class `name` attribute must exactly match the processor
name in the YAML config.
- Default to bridge networking plus explicit `-p` port mappings. Recommend
Docker host-network or host-IPC modes only when the deployment explicitly
needs host namespace sharing, and call out the security tradeoff before
opting in.
- Before running containers, check `nvidia-smi` for free GPU memory. If GPUs
are occupied or low on memory, select a free GPU with `--gpus device=N` or
ask the user to free resources — OOM failures can be silent or misleading.
- Key runtime env vars: `HTTP_PORT` (default 8000), `LOG_LEVEL` (default INFO),
`DEBUG` (0/1), `N_CODEC_INSTANCES` (parallel video decoders),
`MAX_BATCH_SIZE` (live stream frame batching).
- DeepStream backends require a separate `nvdsinfer_config.yaml` in the
model directory.
- When writing new configs, the required top-level fields are only `name`,
`model_repo`, and `models` — everything else is optional.
## Examples
- "Generate an inference-builder YAML config for a DeepStream detection service
that reads RTSP, uses Triton, and exposes a FastAPI endpoint."
- "Adapt the dummy sample into a serverless image-inference pipeline and show
the Docker build and run commands."
Files in this package (4)
- BENCHMARK.md 4 KB
- SKILL.md 6 KB
- skill-card.md 4 KB
- evals/evals.json 3 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · nvidia/deepstream | 100 | Apache-2.0 |
| nvidia-ai-iot/inference_builder | 100 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version e71d84c4c2ca (SHA-256 digest of the entire package)
Downloads in the last 30 days: 1 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100nvidia-deepstream-dev
NVIDIA DeepStream SDK 9.0 development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration. - 100deepstream-import-vision-model
Use this skill to bring a supported object-detection vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF… - 90model-inference-optimize
优化实际模型推理链路,将正确性对齐、分段 profiling、显存与数据搬运、TensorRT/ONNX/PyTorch 后端、attention/kernel、FP8/compile、缓存与少步采样、质量回归、GPU 成本和服务验收串成同一实验闭环。当用户要求推理提速、降低显存或 GPU 成本、复现模型效果、定位 GPU 利用率低、优化图像/视频/扩散模型或自托管 LLM 时使用,提供 model inference performance optimization 的脱敏案例。纯 API 调用、提示词或 token 用量优化,以及电脑卡顿,不进入本技能的模型推理实验流程。 - 100nvidia-deepstream-import-vision-model
Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only. - 100tei-triton-serving
High-performance serving of embedding and reranker models in production. Covers HuggingFace Text Embeddings Inference (TEI) on GPU, Docker deployment, ONNX/FP16 quantization, dynamic batching, NVIDIA Triton for co-serving embedding + reranker + LLM, and NVIDIA NIM as a managed… - 100deepstream-generate-pipeline
Build DeepStream GStreamer pipelines interactively. Use when the user asks about pipelines for video/image inference, detection, tracking, or streaming — including natural phrases like 'pipeline to infer on image', 'run inference on video', 'detect objects in stream', 'save…