tensor-core-utilization
/tensor-core-utilization
Get matrix multiplies onto the tensor cores by fixing shapes, precision, and alignment, then measure that the cores actually fired. Use when a GEMM or attention kernel runs far below the card's advertised throughput and you suspect it fell back to the CUDA cores.
- Author
- Amey-Thakur ↗
- Repository
- AI-SKILLS ↗
- Licence
- MIT
- Last repository change
- 05.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Get matrix multiplies onto the tensor cores by fixing shapes, precision, and alignment, then measure that the cores actually fired. Use when a GEMM or attention kernel runs far below the card's advertised throughput and you suspect it fell back to the CUDA cores.
Original author description
Get matrix multiplies onto the tensor cores by fixing shapes, precision, and alignment, then measure that the cores actually fired. Use when a GEMM or attention kernel runs far below the card's advertised throughput and you suspect it fell back to the CUDA cores.
How it works
- Method
- Checks
- Boundaries
SKILL.md preview
--- name: tensor-core-utilization description: Get matrix multiplies onto the tensor cores by fixing shapes, precision, and alignment, then measure that the cores actually fired. Use when a GEMM or attention kernel runs far below the card's advertised throughput and you suspect it fell back to the CUDA cores. --- # Tensor core utilization Tensor cores perform a small matrix multiply-accumulate per instruction and supply most of a modern GPU's FLOPs, but only when shapes, dtype, and alignment let the library dispatch to them. A GEMM that looks healthy can quietly run on the slower FP32 CUDA cores at a fraction of peak because one dimension is odd or the inputs are the wrong precision. The job is to satisfy the dispatch conditions and then confirm the cores lit up. ## Method 1. **Feed the shapes the MMA units want.** The units work on fixed tiles, so keep M, N, and K multiples of 8 for FP16 and BF16, and multiples of 16 for INT8. On A100 and newer, cuBLAS is happiest when all three are multiples of 16. Pad a dimension like 4095 up rather than leaving it odd. 2. **Use a precision the cores accept.** Tensor cores run FP16, BF16, TF32, INT8, and FP8 on Hopper, not plain FP32. On Ampere and later, enable TF32 (`torch.backends.cuda.matmul.allow_tf32 = True`) so existing FP32 matmuls dispatch to tensor cores with minimal accuracy loss. Prefer BF16 for training because of its wider exponent range. 3. **Align pointers and leading dimensions.** Best throughput needs inputs aligned to 16 bytes, and cuBLASLt prefers leading dimensions that are multiples of 8 elements. Misaligned or non-contiguous tensors force a slower path or extra copies; avoid slicing that produces unaligned views into the matmul. 4. **Let the library choose the kernel, then pin it.** cuBLAS and CUTLASS ship tuned tensor-core kernels. Enable autotuning (`torch.backends.cudnn.benchmark = True` for fixed shapes) so it picks the best tile, and cache that choice for stable shapes instead of re-searching every step. 5. **Measure that tensor cores actually ran.** Do not assume. In Nsight Compute read `sm__pipe_tensor_op_hmma.avg.pct_of_peak_sustained_active`; near zero means the work fell to CUDA cores. Across a cluster, watch the DCGM field `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`. A profiler trace showing kernels named `s16816gemm` or `cutlass...tensorop` also confirms the path. 6. **Raise arithmetic intensity so the cores stay fed.** Tensor cores are fast enough that many GEMMs turn memory-bound. Fuse the epilogue (bias, activation) into the GEMM, batch small matmuls together, and increase tile reuse so HBM bandwidth is not the ceiling that hides the compute you unlocked. ## Checks - Does the tensor-pipe utilization metric sit well above zero for the target kernel? - Are M, N, and K all multiples of 8 (16 for INT8), with any odd dimension padded? - Are the inputs a tensor-core dtype (FP16, BF16, TF32, INT8, FP8), not FP32? - Does measured TFLOP/s approach a real fraction of the card's tensor-core peak, not its FP32 peak? ## Boundaries This covers making matmul-shaped work reach the tensor cores and confirming it. The numerical safety of running in FP16 or BF16 (loss scaling, overflow) is mixed-precision-deployment; INT8 calibration and accuracy gates are quantization-deployment. Shapes fixed by model architecture may resist padding, which is a modeling trade-off, not a kernel fix.
Versions
The database has no other skills named tensor-core-utilization.
Database snapshot: 13.11.2026 · package version 71ed71f00269 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90cuda-cpp-kernels
How to write CUDA C++ kernels for fast-kernel candidates via torch.utils.cpp_extension.load_inline with the pip-installed nvcc - fused epilogues, warp-level reductions, grid barriers / persistent kernels, wmma/mma tensor cores. CUDA C++ is the implementation backend for every… - 100kernel-cute-writing
Write and implement GPU kernels using NVIDIA CuTe DSL (CUTLASS 4.x Python API) — NOT for Triton, CUDA C++, or conceptual explanations. Trigger only when the user wants to write or implement a kernel, not when asking questions about CuTe DSL concepts or layouts. CuTe DSL uses… - 96cuda-streams
Manage tensor lifetimes across CUDA streams and decide whether record_stream is avoidable. Use when using a tensor on a stream other than the one it was allocated on, adding or reviewing Tensor.record_stream / CUDACachingAllocator::recordStream, side streams, wait_stream, or… - 100cuda-attention-kernel-patterns
Patterns and pitfalls for the ONNX domain Attention operator (opset 23/24) CUDA implementation. Use when modifying the dispatch cascade in core/providers/cuda/llm/attention.cc, writing mask/bias CUDA kernels, debugging attention test routing, or adding features to the ONNX… - 90cuda-kernel-basics
Write first CUDA kernels that map work onto the grid, coalesce global memory, and keep enough warps resident to hide latency. Use when hand-writing or reviewing a CUDA kernel and it runs far below the bandwidth or FLOPs the card should reach. - 85kernel-evolution
Decide whether a GPU kernel is the right thing to change, then write, verify and measure one honestly. Use when a profile points at an operator, when choosing between Triton and CUDA, when deciding whether to fuse, when a kernel is fast but the server is not, or when checking…