Przejdź do treści
Biblioteka skilli
Programowanie · Analiza

cuda-profiling

/cuda-profiling

Prowadzi profilowanie CUDA w Nsight Systems i Nsight Compute: metryki roofline i zajętości oraz adnotacje faz przez NVTX.

Licencja
Apache-2.0
Ostatnia zmiana repozytorium
28.09.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
2
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 28.09.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co robi i kiedy

Prowadzi profilowanie CUDA w Nsight Systems i Nsight Compute: metryki roofline i zajętości oraz adnotacje faz przez NVTX.

Oryginalny opis (EN)

Use when profiling CUDA with Nsight Systems or Nsight Compute, reading roofline and occupancy metrics, or annotating phases with NVTX. Not for correctness: use cuda-debugging.

Jak działa

  1. Contract
  2. Inputs
  3. Procedure
  4. Failure and recovery
  5. Output
Sekcje z nagłówków SKILL.md, w oryginale · 116 linii, 887 słów
Podgląd SKILL.md
---
name: cuda-profiling
description: 'Use when profiling CUDA with Nsight Systems or Nsight Compute, reading roofline and occupancy metrics, or annotating phases with NVTX. Not for correctness: use cuda-debugging.'
---

# CUDA profiling

## Contract

| Field | Bound contract |
|---|---|
| Trigger | A CUDA kernel or pipeline is slow or unexplained, and the job is timeline capture, per-kernel metrics, roofline reading, occupancy analysis, or NVTX annotation. |
| Authority | Reversible local. Writes are limited to profiler reports and logs under the project tree; rollback is deleting those files. No remote mutation. |
| Side effect | Report files (`.nsys-rep`, `.ncu-rep`, CSV), possibly elevated-privilege profiling runs, and a diagnosis. |
| Done | The bottleneck is named with a measured metric, and the recommended fix targets that measurement. |

## Inputs

1. Workload (required): the application or kernel to profile and one representative input.
2. Question (required): timeline overlap, per-kernel cost, occupancy, or memory versus compute balance.
3. GPU and toolkit (required): `nvidia-smi` and `nvcc --version`. Grounded current stable: CUDA Toolkit 13.3 Update 1.
4. Profiling permissions (required on locked-down hosts): membership checks in step 7.

## Procedure

1. Pick the tool by question. System-wide timeline across CPU, GPU, and CUDA API calls goes to Nsight Systems (`nsys`). Per-kernel counters and stall analysis goes to Nsight Compute (`ncu`). A single metric in CI goes to `ncu --metrics`. Done when: the tool matches the question.
2. Build with line info and no `-G`:

```bash
nvcc -lineinfo -O3 -arch=sm_80 -o app main.cu
```

Done when: the profiled binary carries line info for source correlation.
3. Capture the timeline first:

```bash
nsys profile --trace=cuda,nvtx,osrt --output=timeline ./app
nsys stats timeline.nsys-rep          # CLI summary
nsys-ui timeline.nsys-rep             # GUI
```

Read the timeline for gaps between launches (CPU bottleneck or sync stalls), `cudaDeviceSynchronize` waits, overlap between copies and kernels across streams, and CUDA API overhead. `--capture-range=cudaProfilerApi` limits capture to a marked region. Done when: each observed gap is attributed to a named cause.
4. Annotate phases with NVTX so timeline bands match application stages:

```cpp
#include <nvtx3/nvtx3.hpp>

nvtxRangePushA("H2D copy");
cudaMemcpyAsync(d_in, h_in, size, cudaMemcpyHostToDevice, stream);
nvtxRangePop();
```

The nvtx3 header-only form is preferred; the C API links with `-lnvToolsExt`. Done when: every phase in the timeline shows as a named band.
5. Analyze the hot kernel in Nsight Compute:

```bash
ncu -o kernel_report ./app
ncu --kernel-name regex:matmul_tiled ./app
ncu --kernel-name regex:hot_kernel --set full ./app
ncu-ui kernel_report.ncu-rep
```

Read the Speed of Light section first: it compares achieved SM and memory throughput against peak. Then read Occupancy, Memory Workload Analysis (hit rates, coalescing), and Warp State Statistics (stall reasons). Done when: the hot kernel's throughput, occupancy, and dominant stall reason are recorded.
6. Collect individual metrics for CI or quick comparison:

```bash
ncu --metrics \
  sm__throughput.avg.pct_of_peak_sustained_elapsed,\
  dram__throughput.avg.pct_of_peak_sustained_elapsed,\
  sm__warps_active.avg.pct_of_peak_sustained_active,\
  l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum,\
  smsp__sass_thread_inst_executed_op_ffma_pred_on.sum \
  ./app

ncu --csv --metrics dram__bytes_read.sum,dram__bytes_write.sum ./app > dram.csv
```

Filter with `--kernel-name` and skip warmup with `--launch-skip` when full sets cost too much. Replay mode controls counter accuracy: `--replay-mode application` replays the whole application per pass. Done when: the metric set answers the stated question and runs in acceptable time.
7. Diagnose memory-bound versus compute-bound from the measurements, not from intuition:

| Reading | Meaning | Fix direction |
|---|---|---|
| `dram__throughput` near peak, `sm__throughput` low | Memory-bound | Coalescing, shared memory tiling, fewer bytes moved |
| `sm__throughput` near peak, `dram__throughput` low | Compute-bound | Tensor cores, unrolling, more ILP |
| Both low | Launch config, occupancy, or sync overhead | Block size, grid size, barrier audit |

Roofline reading: arithmetic intensity in FLOP per byte against the machine's balance point. Approximate single-precision intensity as `smsp__sass_thread_inst_executed_op_ffma_pred_on.sum * 2 / dram__bytes.sum`; treat it as a lower bound, since it counts FMA instructions only. Done when: the kernel is classified with the two throughput numbers attached.
8. Attribute occupancy limits before tuning:

```bash
ncu --metrics sm__warps_active.avg.pct_of_peak_sustained_active,\
launch__occupancy_limit_registers,\
launch__occupancy_limit_shared_mem,\
launch__occupancy_limit_block_size ./app
```

| Limiting factor | Typical fix |
|---|---|
| Registers | `-maxrregcount`, or simplify the kernel |
| Shared memory | Smaller tile, split phases |
| Block size | Try 128 or 256 instead of 512+ |

Done when: the binding limit is named and the fix addresses it.
9. On permission errors, fix the environment before re-running. `ERR_NVGPUCTRPERM` means the driver blocks non-admin counters; the module parameter `NVreg_RestrictProfilingToAdminUsers=0` lifts it, or the run goes under sudo. Verify the intended GPU is visible through `CUDA_VISIBLE_DEVICES`. Done when: the counter access error is gone or the constraint is reported to the user.

## Failure and recovery

| Failure class | Behavior |
|---|---|
| `ERR_NVGPUCTRPERM` | Profiling permission missing. Apply step 9 or report the needed driver setting. |
| All metrics read zero | Wrong GPU visible or profiling disabled. Check `CUDA_VISIBLE_DEVICES`; use `--target-processes all` for child processes. |
| NCU report empty | Kernel too short or never launched. Enlarge the workload; verify `cudaGetLastError()`. |
| Profiling overhead distorts behavior | Full metric sets on many kernels. Filter with `--kernel-name`, skip with `--launch-skip`. |
| Timeline shows no overlap | Single default stream. Introduce streams and async copies, then recapture. |
| Occupancy high but kernel slow | Latency is not hidden or access is uncoalesced. Cross-check with gpu-memory-model and recollect stall reasons. |

## Output

A diagnosis: the classified bottleneck, the metrics that support it, the timeline or kernel report path, and the fix ranked by expected effect. Report files stay in the project tree for comparison runs; name them per variant so before-and-after CSVs stay aligned.
Pliki w paczce (2)
  • SKILL.md 7 KB
  • agents/openai.yaml 1 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · OutlineDriven/outline-driven-development100Apache-2.0
mohitmishra786/low-level-dev-skills100MIT

Migawka bazy z 13.11.2026 · wersja paczki e7e5524e9051 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100kernel-profiling-nsight
    Profiluje pracę GPU w Nsight Systems i Nsight Compute: czyta oś czasu, nazywa klasę wąskiego gardła i wskazuje metrykę, która określa poprawkę.
  • 100perf-nsight-systems
    Obsługuje CLI Nsight Systems (nsys) do profilowania osi czasu systemu: profile, raporty .nsys-rep, nsys stats, bezczynność GPU i trening rozproszony z nakładaniem NCCL.
  • 100nvidia-tensorrt-llm-perf-nsight-systems
    Obsługuje CLI Nsight Systems (nsys) do profilowania osi czasu: nsys profile, analiza raportów .nsys-rep, nsys stats, diagnoza bezczynności GPU i nakładania NCCL w treningu rozproszonym.
  • 100nsys-optimizer
    Optymalizuje kod symulacji CUDA i GPU na podstawie profilowania NVIDIA Nsight Systems, wskazuje wąskie gardła i prowadzi całą pętlę optymalizacji, na przykład gdy jedna scena działa wolniej.
  • 100nvtx-python
    Profiluje funkcje Pythona z NVTX poza Kit (Isaac Lab 3.0+ standalone i inne aplikacje bez Carbonite) przez sitecustomize.py z hakiem sys.setprofile i integracją z Nsight Systems.
  • 90nvidia-cuda-kernel-performance-review
    Przegląda statycznie źródła kerneli CUDA C/C++ względem opublikowanych wskazówek wydajności NVIDIA: koalescencja pamięci globalnej, konflikty banków pamięci współdzielonej, rozbieżność warpów, occupancy, strumienie i parametry uruchomienia.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.