Skip to content
Skills Library
AI & agents · Building & coding

tensor-parallel

/tensor-parallel

Trace a model too big for one GPU by sharding it with transformers tensor parallelism — TransformersModel(..., distributed_config=DistributedConfig(tp_size=N)) under torchrun, with the full API (gradients, .source, list batches); for throughput, serving or streaming without those, see the vllm skill. Use when a checkpoint does not fit on one card, for multi-GPU tracing, tp_size or tp_plan, or activation shapes a fraction of the expected width (hidden_size/N). Covers the rules SPMD puts on intervention code (no rank-dependent control flow, seed before sampling, clone before editing any value a

Repository
skills ↗
Licence
MIT
Last repository change
19.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
7
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 19.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • SKILL.md zawiera komentarze HTML (7). Nie widać ich w podglądzie Markdown, a model je czyta.

What it does and when to use it

Trace a model too big for one GPU by sharding it with transformers tensor parallelism — TransformersModel(..., distributed_config=DistributedConfig(tp_size=N)) under torchrun, with the full API (gradients, .source, list batches); for throughput, serving or streaming without those, see the vllm skill. Use when a checkpoint does not fit on one card, for multi-GPU tracing, tp_size or tp_plan, or activation shapes a fraction of the expected width (hidden_size/N). Covers the rules SPMD puts on intervention code (no rank-dependent control flow, seed before sampling, clone before editing any value a

Original author description

Trace a model too big for one GPU by sharding it with transformers tensor parallelism — TransformersModel(..., distributed_config=DistributedConfig(tp_size=N)) under torchrun, with the full API (gradients, .source, list batches); for throughput, serving or streaming without those, see the vllm skill. Use when a checkpoint does not fit on one card, for multi-GPU tracing, tp_size or tp_plan, or activation shapes a fraction of the expected width (hidden_size/N). Covers the rules SPMD puts on intervention code (no rank-dependent control flow, seed before sampling, clone before editing any value a

How it works

  1. Loading
  2. Reading and editing
  3. What is actually sharded
  4. Values between two sharded modules
  5. Parameters are DTensors
  6. The three rules for intervention code
  7. Requires transformers >= 5.19
  8. Expert parallelism
  9. Not supported
  10. Diagnosing
Original SKILL.md headings · 276 lines, 2,010 words
SKILL.md preview
---
name: tensor-parallel
description: Trace a model too big for one GPU by sharding it with transformers tensor parallelism — TransformersModel(..., distributed_config=DistributedConfig(tp_size=N)) under torchrun, with the full API (gradients, .source, list batches); for throughput, serving or streaming without those, see the vllm skill. Use when a checkpoint does not fit on one card, for multi-GPU tracing, tp_size or tp_plan, or activation shapes a fraction of the expected width (hidden_size/N). Covers the rules SPMD puts on intervention code (no rank-dependent control flow, seed before sampling, clone before editing any value a sharded module produced), the transformers >= 5.19 requirement, sharded weights as DTensors, and what is not supported.
---

# Tensor Parallelism

Transformers tensor parallelism splits every attention and MLP projection *within*
each layer across GPUs. nnsight gathers sharded activations before your
intervention sees them and re-splits what you leave behind, so **the trace you
write is the trace you would write against one GPU**. (`device_map="auto"`, which
puts whole layers on different GPUs, needs nothing from this skill.)

> Not executed in CI: needs ≥2 GPUs and torchrun. The `max_tp_size` block below
> is config-only and does run.

## Loading

One process per GPU under `torchrun`. **Every rank runs the whole script,
including your intervention code.**

<!-- test: skip -->
```python
# tp_trace.py  —  torchrun --nproc_per_node=4 tp_trace.py
import torch
from transformers.distributed import DistributedConfig
from nnsight.modeling.transformers import TransformersModel

model = TransformersModel(
    "meta-llama/Llama-3.2-3B",
    task="text-generation",
    dispatch=True,
    dtype=torch.bfloat16,
    distributed_config=DistributedConfig(tp_size=4),
)
```

`tp_plan="auto"` is degree-checked against the world size the same way; a custom
plan (a dict of patterns) loads unchecked. Prefer `DistributedConfig(tp_size=N)`:
it states the degree, and it is the only form that can ask for expert parallelism.

`tp_size` must divide the model's attention heads, key/value heads, and
intermediate size. Check before you allocate anything — this reads the config
alone, so it needs no GPU, no weights and no `torchrun`:

<!-- test: setup -->
```python
from transformers import AutoConfig
from nnsight.modeling.tp import max_tp_size

assert max_tp_size(AutoConfig.from_pretrained("Qwen/Qwen2.5-0.5B")) == 2
assert max_tp_size(AutoConfig.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")) == 3
assert max_tp_size(AutoConfig.from_pretrained("openai-community/gpt2")) is None
```

Both stop at their key/value head count; every workable degree divides it.
`None` means the checkpoint cannot be split at all — no `base_model_tp_plan`
(gpt2, and several recent Qwen releases), or a refused style. Asking anyway raises
`UnshardableCheckpoint` naming the degrees that would have worked.

## Reading and editing

Reading is identical to single-GPU code: `gate_proj.output` arrives at its full
`(1, seq, 8192)` even though each rank computed a slice of it.

**Editing takes a clone — on any value a sharded module produced.** Whether
nnsight gathered it or transformers all-reduced it, the whole value you see is the
output of a collective, and torch refuses an in-place write into one. That covers
the gathered outputs and also `o_proj.output`, `down_proj.output`, `mlp.output`,
the tensor in `self_attn.output`, an MoE's `experts.output` and `lm_head.output` —
all of which take in-place writes on one GPU. `gate_proj.output[..., :3000] = 0`
raises `RuntimeError: Output 0 of SliceBackward0 is a view and is being modified
inplace`. Clone, edit, assign back:

<!-- test: skip -->
```python
gate_proj = model.model.layers[5].mlp.gate_proj

with model.trace(prompt):
    edited = gate_proj.output.clone()
    edited[..., :3000] = 0
    gate_proj.output = edited
    logits = model.lm_head.output.save()
```

Replacing the whole value (`gate_proj.output = torch.zeros_like(...)`) needs no
clone. nnsight does not clone for you: on a large model that is a copy the size
of the activation, on every gather, for the many traces that only read.

## What is actually sharded

| Value | Arrives |
|---|---|
| Column-parallel **output** (`q_proj`, `k_proj`, `v_proj`, `gate_proj`, `up_proj`) | sharded — gathered for you |
| Row-parallel **input** (`o_proj.input`, `down_proj.input`) | sharded — gathered for you |
| Row-parallel output (`o_proj.output`, `down_proj.output`) | whole by all-reduce — clone to edit |
| Whole modules (`layers[i].output`, `self_attn`, `mlp.output`, `norm.output`) | whole — clone to edit `self_attn`/`mlp` |
| `lm_head.output` | whole, gathered by transformers — clone to edit |
| **Parameters** (`q_proj.weight`) | **sharded, not gathered** — see below |
| Values *between* two sharded modules (`mlp.act_fn.output`, `query_states_0`) | **sharded, not gathered** — see below |
| `embed_tokens.output` | whole — unless the plan shards it, and then not readable, see below |

The gather only fires where an intervention is parked, so reading a few
locations does not pay for the hundreds you ignore.

**A vocab-parallel embedding's own output cannot be read.** Plans that shard the
embedding name it `embedding_rowwise` — in practice the tied-embedding checkpoints
(Llama-3.2-1B and -3B, most small models). There, parking on `embed_tokens.output`
raises a bare `AssertionError` from torch's embedding op, and so does
`tracer.cache()` with *no* arguments, which reaches the embedding. Check with
`"embed_tokens" in (model.config.base_model_tp_plan or {})`. Read
`model.model.layers[0].input` instead — the same tensor one module later, whole at
any degree — and name the modules you cache.

## Values between two sharded modules

A module's own `.input`/`.output` is made whole; a value *between* two sharded
modules is not, and nothing on it records which axis holds the shard (attention's
`view`/`transpose` moves it to the head dimension). Name the axis yourself:

<!-- test: skip -->
```python
from nnsight.modeling.tp import gather, shard

attn = model.model.layers[1].self_attn

with model.trace(prompt):
    q = attn.source.query_states_0.output       # (1, heads/tp_size, seq, head_dim)
    heads = gather(model, q, dim=1).clone()     # (1, heads, seq, head_dim)
    heads[:, 3] = 0                             # ablate head 3, whoever holds it
    attn.source.query_states_0.output = shard(model, heads, dim=1)
    logits = model.lm_head.output.save()
```

Both are **collectives**: every rank must reach them (rule 1), so call them
unconditionally. Both are no-ops on an unsharded model.

**A one-GPU run validates nothing.** At `tp_size=1` the value comes back
untouched, so a wrong `dim` is accepted silently and only raises (or garbles the
tensor) on a sharded run; an in-place edit with no clone works at `tp_size=1` and
fails under TP. Smoke-test a gather/shard block at `tp_size=2` before trusting it.

A `.source` value from *inside* a sharded module is a `DTensor`: its
`.shape` reports the **whole** while its data is this rank's slice, so
`float(v.sum())` is quietly wrong and comparing widths across degrees shows
nothing. Test with `isinstance(v, DTensor)` and reassemble with
`gather(model, v, dim=...)` or `v.full_tensor()`.

## Parameters are DTensors

`layer.weight` is a `DTensor`: this rank holds `1/tp_size` of it, but **`.shape`
reports the whole**, so shape alone will not tell you it is split. Llama-3.2-3B
at tp=2:

<!-- test: skip -->
```python
w = model.model.layers[0].self_attn.q_proj.weight
w.shape                 # (3072, 3072) — the global shape
w.placements            # (Shard(dim=0),)
w.to_local().shape      # (1536, 3072) — what this rank holds
w.full_tensor()         # the whole, allocated on every rank (a collective)
```

**A reduction over a sharded weight returns this rank's answer, with no error.**
`w.mean()`, `w.norm()`, `w.abs().max()` return a `Partial` `DTensor`, and `float()`
reads this rank's uncombined number:

<!-- test: skip -->
```python
float(w.mean())                  # rank 0:  2.256e-05   rank 1: -1.171e-05
float(w.mean().full_tensor())    # 5.421e-06 on both — the real mean
float(w.norm())                  # rank 0: 81.80        rank 1: 68.97
float(w.norm().full_tensor())    # 106.996
```

Call `.full_tensor()` on the **reduction**, not the weight — one small collective
rather than a copy of the layer — or a weight-norm sweep plots per-rank noise.

**A sharded weight cannot be edited in place through the `DTensor`.**
`weight[:, :10] = 0` raises `NotImplementedError: Operator aten.fill_.Tensor does
not have a sharding strategy registered`. Write through `.to_local()` under
`torch.no_grad()` — each rank edits the rows or columns it holds. A rank-one
(ROME-style) update has to be expressed per-rank the same way; the
`model-editing-and-lora` skill has the single-GPU form.

## The three rules for intervention code

Every rank runs your block — that is what keeps the collectives lined up.

**1. No rank-dependent control flow.** Nothing may branch on rank or take a
different path on different ranks; the ranks stop agreeing on when to gather and
the run deadlocks — quietly, with no exception and no watchdog for minutes. A
plain `kill` (SIGTERM) of `torchrun` takes the ranks down with it; after
`kill -9` they survive holding the cards, so find the rank pids and kill those.

**2. Seed before you sample.** `torch.initial_seed()` differs per rank under
`torchrun`, so unseeded sampling makes the ranks generate different tokens, and
the model's all-reduces then sum activations from different sequences: the output
is *wrong* on every rank, not merely different. Many checkpoints ship
`do_sample: true` in `generation_config.json`, so it bites without asking. Pass
`do_sample=False`, or call `torch.manual_seed(0)` on every rank before generating.

**3. Clone before editing any value a sharded module produced** (see "Reading
and editing").

Every rank computes the **same saved values**, so print or write results from one
rank (`if int(os.environ.get("RANK", 0)) == 0: print(...)`) or you get N copies.
That `if` goes around the *print*, outside the trace — the one place a rank test is
safe. Never inside a block.

## Requires transformers >= 5.19

nnsight reads the exact layout of 5.19's distributed backend, and a layout it
cannot read would hand your code one rank's slice as though it were the whole
tensor. So 5.16–5.18 are refused with `UnsupportedTransformersVersion` the moment
a sharded model is seen; releases before the DTensor backend (≤ 5.15) stop earlier,
at `ModuleNotFoundError: No module named 'transformers.distributed.tensor_parallel'`.

## Expert parallelism

A mixture-of-experts checkpoint can be split by *expert* instead of along the
feature dimension:

<!-- test: skip -->
```python
model = TransformersModel(
    "Qwen/Qwen3-30B-A3B",
    task="text-generation",
    distributed_config=DistributedConfig(tp_size=2, ep_size=2),
    dispatch=True,
)
```

`mlp.experts.output` and `mlp.gate.output` come back at their single-GPU shapes
and values, and edits to them (cloned, rule 3) carry through. The degree has to
divide the **expert count**; pre-flight with
`max_tp_size(config, expert_parallel=True)`, which reads `base_model_ep_plan`.
Plain tensor parallelism on an MoE checkpoint is also fine: the experts shard
along their feature dimensions and behave like any MLP.

## Not supported

Four styles are refused: `megamoe_router`, `megamoe_experts`,
`moe_identity_expert`, and MLA's split kv projection (`mla_kv_a_proj`). A plan
containing one supports no degree, so the load raises `UnshardableCheckpoint`.
DeepSeek-V2-Lite is the one to know: `mla_kv_a_proj` rules out tensor
parallelism, while expert parallelism is still open to it.

## Diagnosing

Three of these arrive wrapped in a pipeline `ValueError: Could not load model
...`, with the real message near the end of it.

| Symptom | Cause |
|---|---|
| `KeyError: 'RANK'` | Started with `python`, not `torchrun`. TP needs the calling process to be a rank, which is also why it cannot run in a notebook kernel |
| `tp_size (2) * fsdp_size (1) is not equal to world_size (4)` | `--nproc_per_node` must equal `tp_size` |
| `` `tp_plan` and `device_map` are mutually exclusive `` | Pick one — they are two different ways of spreading a model |
| `UnshardableCheckpoint` | No `base_model_tp_plan`, a refused style, or a degree that divides nothing. The message lists the degrees that work. Load on one GPU or with `device_map` |
| `UnsupportedTransformersVersion` | transformers 5.16–5.18 — upgrade to 5.19, or load on one GPU |
| `ModuleNotFoundError: transformers.distributed.tensor_parallel` | transformers ≤ 5.15 — upgrade, or load on one GPU |
| Nothing raised, but `fragments.enabled` is `False` and every rank holds the whole model | A custom `tp_plan` dict whose patterns match nothing — a dict plan is not degree-checked |
| Activation width is `hidden_size/N` or `intermediate_size/N` | A value between two sharded modules. `gather(model, v, dim=...)`, naming the axis |
| Shape looks right but the numbers are wrong, and differ per rank | A `DTensor` reporting the global shape: a `.source` value inside a sharded module, or a reduction over a weight. Check `v.placements`; fix with `.full_tensor()` |
| Bare `AssertionError`, empty message | `embed_tokens.output` on a plan that shards the embedding, or a `tracer.cache()` with no arguments that reaches it. Read `layers[0].input` and name the modules to cache |
| `RuntimeError: Output 0 of ... is a view and is being modified inplace` | Editing a value a sharded module produced in place — clone, edit, assign back |
| `NotImplementedError: Operator aten.fill_.Tensor does not have a sharding strategy` | Editing a sharded weight in place — write through `.to_local()` |
| Run hangs with no error | The ranks diverged — rank-dependent control flow, or an exception on one rank only. `kill` torchrun; after a `kill -9`, kill the rank pids too |
| Generated text differs per rank | Sampling without an identical seed (rule 2) |
| Results differ from single-GPU by ~1e-3 in bfloat16 | Expected: an all-reduce sums in a different order than one matmul. In float32 the same gap is ~1e-6; anything larger there is a real difference. Greedy token choices are identical either way |

## Choosing between this and vLLM

Both shard across GPUs. Use **tensor parallelism** when you want ordinary
`TransformersModel` semantics — gradients, `.source`, `tracer.result`, list
inputs — on a model that does not fit. Use **vLLM** (the `vllm` skill, with
`tensor_parallel_size`) when you need throughput, continuous batching, serving or
async streaming, and can accept its differences (no gradients, one prompt per
invoke, `model.logits` / `model.samples`).

Versions

The database has no other skills named tensor-parallel.

Database snapshot: 13.11.2026 · package version bcbe9c1a8cc1 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90vllm-serving
    Deploy an LLM on vLLM with continuous batching, deliberate VRAM planning, and multi-model hosting that never overcommits the card. Use when serving on vLLM and deciding memory fraction, context length, tensor parallel size, and how many models share a GPU.
  • 100vllm
    Run nnsight interventions inside the vLLM inference engine — continuous batching, tensor parallelism, CUDA-graph taps, async streaming, engine-wide edit() and nnsight-serve. Use for throughput, many concurrent prompts, streamed tokens, or a served engine; for a sharded model…
  • 100vllm
    Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), OpenAI-compatible API serving, throughput and latency benchmarking, continuous batching tuning, GPU…
  • 100model-parallelism
    Split a model that will not fit on one GPU across devices with tensor, pipeline, and expert parallelism, choosing each split by its communication cost. Use when weights or activations exceed a single card and you must shard without stalling on the interconnect.
  • 100distributed-training
    Use this skill when the user says 'distributed training', 'multi-gpu training', 'DDP', 'FSDP', 'ZeRO', 'DeepSpeed', 'all-reduce', 'tensor parallel', 'pipeline parallel', 'torchrun', 'NCCL'. Chooses and configures a parallelism strategy (data, sharded data, tensor, pipeline,…
  • 95training-llms-megatron
    Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism.…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.