Skip to content
Skills Library
AI & agents · Building & coding

inference-builder

/inference-builder

Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.

Repository
deepstream ↗
Licence
Apache-2.0
Last repository change
22.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
4
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 22.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.

Original author description

Generate deployable Vision AI pipelines with high-performance video and streaming capabilities on NVIDIA GPUs. Activate when users want to: create GPU-accelerated inference microservices or standalone apps for vision, video, or streaming workloads; write or edit pipeline YAML configs; build Docker images for GPU inference; work with models from NGC or HuggingFace; or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.

How it works

  1. Setup
  2. Key Directories
  3. Instructions
  4. Non-Obvious Knowledge
  5. Examples
Original SKILL.md headings · 116 lines, 747 words
SKILL.md preview
---
name: inference-builder
description: |
  Generate deployable Vision AI pipelines with high-performance video and
  streaming capabilities on NVIDIA GPUs. Activate when users want to: create
  GPU-accelerated inference microservices or standalone apps for vision,
  video, or streaming workloads; write or edit pipeline YAML configs; build
  Docker images for GPU inference; work with models from NGC or HuggingFace;
  or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.
owner: "chunlinl@nvidia.com"
metadata:
  author: "Chunlin Li <chunlinl@nvidia.com>"
  version: "1.0.0"
  tags:
    - inference
    - nvidia
    - deepstream
    - vllm
    - tensorrt-llm
    - triton
    - docker
    - gpu
service: "inference"
reviewed: "2026-04-09"
---

# NVIDIA Inference Builder

A code generator for Vision AI pipelines with high-performance video and
streaming capabilities on NVIDIA GPUs. It turns a YAML config (+ optional
OpenAPI spec and custom processors) into a deployable inference pipeline,
packaged as a Docker container or standalone application.

## Setup

Inference Builder is part of the DeepStream mono-repo. Resolve it from the
repository root so the skill works from any subdirectory:

```bash
DEEPSTREAM_ROOT="$(git rev-parse --show-toplevel)"
INFERENCE_BUILDER_ROOT="$DEEPSTREAM_ROOT/tools/inference_builder"
source "$INFERENCE_BUILDER_ROOT/.venv/bin/activate"
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
```

### First-time venv setup

```bash
cd "$INFERENCE_BUILDER_ROOT"
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### Getting current CLI usage

Always run with `--help` for authoritative flags — do NOT rely on memorized
syntax:

```bash
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
```

All paths below are relative to `tools/inference_builder/` unless stated otherwise.

## Key Directories

| Path | What lives there |
|------|-----------------|
| `builder/samples/` | Working examples for every backend — read `builder/samples/README.md` for the full index, then start from the closest matching example |
| `schemas/config.schema.json` | Main config schema — all fields, data types, and validation rules |
| `schemas/README.md` | Comprehensive reference: backends, parameters, data types, preprocessors, responders, routing |
| `schemas/backends/` | Per-backend schemas and parameter definitions |
| `templates/backend/` | Jinja templates for each inference backend |
| `templates/responder/` | Jinja templates for server endpoint handlers |
| `lib/` | Shared runtime libraries included in generated pipelines |
| `builder/tests/` | Integration test framework (see `run_tests.sh --help`) |

## Instructions

1. **Browse samples & schemas** — find the closest match in `builder/samples/`;
   consult `schemas/config.schema.json` and `schemas/backends/` for field reference
2. **Write YAML config** — validate against the schema; consult
   `schemas/README.md` for data types, preprocessors, responders, and routing.
   Prefer choices that keep the hot path on GPU and skip avoidable GPU↔CPU
   round-trips; prefer GPU acceleration across the whole pipeline — decode/
   encode, scale & color convert, batch, preprocess, infer, track, and overlay
3. **Generate pipeline** — `python builder/main.py <config.yaml> [options]`
4. **Prepare model repository** — download models from NGC/HuggingFace into
   the model_repo directory
5. **Generate nvinfer config** (DeepStream only) — check if the downloaded
   model already includes runtime config files (nvdsinfer_config.yaml,
   nvdspreprocess_config.yaml); if not, generate them and place alongside
   the model. Skip this step entirely for non-DeepStream backends
6. **Select platform & build Docker image** — read `doc/platform-guide.md` to
   choose the correct base image and Dockerfile for the target hardware
   (x86_64, Jetson/Tegra, DGX Spark); then build the image
7. **Run & test** — run the container with the model repository mounted;
   for serverless pipelines the container runs to completion; for HTTP
   servers (fastapi/triton/nim) send test requests to verify. If GPU
   resources are available, perform an end-to-end smoke test against the
   running container to validate the full pipeline on real hardware

## Non-Obvious Knowledge

- If the user's model request is ambiguous (e.g., unclear model variant,
  version, precision, or source), always ask for clarification before
  proceeding — do not guess.
- Before using any model, gather as much information as possible about it:
  version, architecture, input/output shapes, precision, and expected
  preprocessing. Cross-check this information against the pipeline YAML
  config and any runtime config (e.g., nvdsinfer_config.yaml) to ensure
  consistency — mismatches in input dims, scale factors, or label files
  are common sources of silent failures.
- Config YAML supports runtime env-var substitution: `$ENV_VAR|default_value`
  (auto-typed based on the default).
- Server type `serverless` does NOT need an OpenAPI spec or server definition.
- Custom processor class `name` attribute must exactly match the processor
  name in the YAML config.
- Default to bridge networking plus explicit `-p` port mappings. Recommend
  Docker host-network or host-IPC modes only when the deployment explicitly
  needs host namespace sharing, and call out the security tradeoff before
  opting in.
- Before running containers, check `nvidia-smi` for free GPU memory. If GPUs
  are occupied or low on memory, select a free GPU with `--gpus device=N` or
  ask the user to free resources — OOM failures can be silent or misleading.
- Key runtime env vars: `HTTP_PORT` (default 8000), `LOG_LEVEL` (default INFO),
  `DEBUG` (0/1), `N_CODEC_INSTANCES` (parallel video decoders),
  `MAX_BATCH_SIZE` (live stream frame batching).
- DeepStream backends require a separate `nvdsinfer_config.yaml` in the
  model directory.
- When writing new configs, the required top-level fields are only `name`,
  `model_repo`, and `models` — everything else is optional.

## Examples

- "Generate an inference-builder YAML config for a DeepStream detection service
  that reads RTSP, uses Triton, and exposes a FastAPI endpoint."
- "Adapt the dummy sample into a serverless image-inference pipeline and show
  the Docker build and run commands."
Files in this package (4)
  • BENCHMARK.md 4 KB
  • SKILL.md 6 KB
  • skill-card.md 4 KB
  • evals/evals.json 3 KB

Versions

RepositoryScoreLicence
This page · nvidia/deepstream100Apache-2.0
nvidia-ai-iot/inference_builder100Apache-2.0

Database snapshot: 13.11.2026 · package version e71d84c4c2ca (SHA-256 digest of the entire package)

Downloads in the last 30 days: 1 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100nvidia-deepstream-dev
    NVIDIA DeepStream SDK 9.0 development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.
  • 100deepstream-import-vision-model
    Use this skill to bring a supported object-detection vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF…
  • 90model-inference-optimize
    优化实际模型推理链路,将正确性对齐、分段 profiling、显存与数据搬运、TensorRT/ONNX/PyTorch 后端、attention/kernel、FP8/compile、缓存与少步采样、质量回归、GPU 成本和服务验收串成同一实验闭环。当用户要求推理提速、降低显存或 GPU 成本、复现模型效果、定位 GPU 利用率低、优化图像/视频/扩散模型或自托管 LLM 时使用,提供 model inference performance optimization 的脱敏案例。纯 API 调用、提示词或 token 用量优化,以及电脑卡顿,不进入本技能的模型推理实验流程。
  • 100nvidia-deepstream-import-vision-model
    Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.
  • 100tei-triton-serving
    High-performance serving of embedding and reranker models in production. Covers HuggingFace Text Embeddings Inference (TEI) on GPU, Docker deployment, ONNX/FP16 quantization, dynamic batching, NVIDIA Triton for co-serving embedding + reranker + LLM, and NVIDIA NIM as a managed…
  • 100deepstream-generate-pipeline
    Build DeepStream GStreamer pipelines interactively. Use when the user asks about pipelines for video/image inference, detection, tracking, or streaming — including natural phrases like 'pipeline to infer on image', 'run inference on video', 'detect objects in stream', 'save…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.