Przejdź do treści
Biblioteka skilli
AI i agenci · Budowanie i kod

distributed-training

/distributed-training

Dobiera i konfiguruje strategię równoległego trenowania modeli w PyTorch, np. DDP, FSDP, ZeRO lub pipeline, i szuka wąskich gardeł komunikacji.

Repozytorium
agent-skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
10.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 10.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Dobiera i konfiguruje strategię równoległego trenowania modeli w PyTorch, np. DDP, FSDP, ZeRO lub pipeline, i szuka wąskich gardeł komunikacji.

Wyzwalacze z opisu

distributed trainingmulti-gpu trainingDDPFSDPZeRODeepSpeedall-reducetensor parallel
Oryginalny opis (EN)

Use this skill when the user says 'distributed training', 'multi-gpu training', 'DDP', 'FSDP', 'ZeRO', 'DeepSpeed', 'all-reduce', 'tensor parallel', 'pipeline parallel', 'torchrun', 'NCCL'. Chooses and configures a parallelism strategy (data, sharded data, tensor, pipeline, context) for PyTorch training, sizes memory, and debugs collective-communication performance. Do NOT use for: inference serving (paged-attention, tensorrt-optimization), network fabric internals (rdma-roce), single-kernel optimization (cuda-programming), experiment tracking (ml-experiment-tracking).

Jak działa

  1. Purpose
  2. Agent Protocol
  3. Memory Model
  4. Choosing a Strategy
  5. Collectives
  6. PyTorch DDP
  7. PyTorch FSDP2 (fullyshard)
  8. DeepSpeed ZeRO Config
  9. Diagnostics
  10. Anti-Patterns
Sekcje z nagłówków SKILL.md, w oryginale · 149 linii, 845 słów
Podgląd SKILL.md
---
name: distributed-training
description: >
  Use this skill when the user says 'distributed training', 'multi-gpu training', 'DDP', 'FSDP', 'ZeRO', 'DeepSpeed', 'all-reduce', 'tensor parallel', 'pipeline parallel', 'torchrun', 'NCCL'. Chooses and configures a parallelism strategy (data, sharded data, tensor, pipeline, context) for PyTorch training, sizes memory, and debugs collective-communication performance. Do NOT use for: inference serving (paged-attention, tensorrt-optimization), network fabric internals (rdma-roce), single-kernel optimization (cuda-programming), experiment tracking (ml-experiment-tracking).
version: "1.0.0"
author: "j4flmao"
license: "MIT"
tags: [hardware-acceleration, distributed-training, pytorch, fsdp, deepspeed, nccl]
---

# Distributed Training

## Purpose
Train models that are too slow or too large for one GPU by splitting data, optimizer state, parameters, or layers across devices. The right strategy follows from a memory budget and the interconnect topology; communication cost, not FLOPs, usually decides scaling efficiency.

## Agent Protocol

### Trigger
"distributed training", "multi-gpu", "multi-node", "DDP", "FSDP", "ZeRO", "DeepSpeed", "Megatron", "all-reduce", "tensor parallel", "pipeline parallel", "torchrun", "NCCL".

### Input Context
- Model size (parameters), sequence length, micro-batch size, precision.
- GPU type/memory, GPUs per node, node count, intra-node (NVLink/PCIe) and inter-node (InfiniBand/RoCE) links.
- Framework (PyTorch native, DeepSpeed, Megatron-Core, Accelerate, Lightning).

### Output Artifact
Parallelism plan + launch command + config. No file unless requested.

### Response Format
```
Memory/GPU: params {GB} + grads {GB} + optimizer {GB} + activations {GB}
Strategy: {DDP|FSDP/ZeRO-n|TP=x PP=y DP=z}
Launch: {torchrun/deepspeed command}
Watch: {MFU, step time, comm/compute overlap}
```

### Completion Criteria
- [ ] Per-GPU memory estimate fits with headroom.
- [ ] Global batch = micro_batch x grad_accum x data_parallel_size documented.
- [ ] Checkpointing works with the sharding scheme (sharded/distributed checkpoint).
- [ ] Scaling efficiency measured (throughput at 1 node vs N nodes).

## Memory Model
Mixed-precision Adam needs ~16 bytes/parameter before activations: 2 (BF16 weights) + 2 (BF16 grads) + 12 (FP32 master weights, momentum, variance). A 7B model therefore needs ~112 GB of model state, more than one 80 GB GPU.

| Scheme | Model-state bytes/param per GPU (N data-parallel ranks) |
|--------|-----------------------------------------------------|
| DDP | 16 |
| ZeRO-1 (shard optimizer) | 4 + 12/N |
| ZeRO-2 (+ shard gradients) | 2 + 14/N |
| ZeRO-3 / FSDP full shard (+ shard params) | 16/N |

Activations scale with micro-batch x sequence x hidden x layers; reduce with activation checkpointing (recompute) and smaller micro-batches.

## Choosing a Strategy
```
Model state fits on one GPU?
├── Yes → DDP (simplest, fastest per-byte)
└── No → FSDP / ZeRO-3 within and across nodes
          Still too large or too slow (layer > GPU, poor overlap)?
          └── Add tensor parallel (TP) inside a node (needs NVLink)
              + pipeline parallel (PP) across nodes
              + context/sequence parallel for very long sequences
```
Rule of thumb: TP traffic is per-layer and latency-sensitive, so keep TP within an NVLink domain; DP/FSDP traffic tolerates the slower inter-node fabric.

## Collectives
- **All-reduce (DDP)**: ring algorithm sends 2(N-1)/N x size bytes per GPU, independent of N in the limit. NCCL also uses tree and NVLS (NVLink SHARP) algorithms depending on topology.
- **Reduce-scatter + all-gather (FSDP/ZeRO-3)**: params all-gathered before each layer's forward/backward, grads reduce-scattered after backward; same total volume as all-reduce plus the param gathers (~1.5x DDP traffic).
- Overlap is everything: DDP buckets gradients (`bucket_cap_mb`) and all-reduces during backward; FSDP prefetches the next layer's params.

## PyTorch DDP
```python
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler

def main() -> None:
    dist.init_process_group(backend="nccl")
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)

    model = build_model().cuda(local_rank)          # build_model: user-defined
    model = DDP(model, device_ids=[local_rank])
    sampler = DistributedSampler(dataset, shuffle=True)  # dataset: user-defined
    loader = DataLoader(dataset, batch_size=8, sampler=sampler, num_workers=4, pin_memory=True)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-4)

    for epoch in range(10):
        sampler.set_epoch(epoch)                     # different shuffle each epoch
        for batch in loader:
            with torch.autocast("cuda", dtype=torch.bfloat16):
                loss = model(**batch).loss
            loss.backward()
            opt.step()
            opt.zero_grad(set_to_none=True)
    dist.destroy_process_group()

if __name__ == "__main__":
    main()
```
```bash
# single node, 8 GPUs
torchrun --standalone --nproc_per_node=8 train.py
# multi node (run on every node)
torchrun --nnodes=4 --nproc_per_node=8 --rdzv_backend=c10d --rdzv_endpoint=$HEAD:29500 train.py
```

## PyTorch FSDP2 (`fully_shard`)
```python
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy

mp = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32)
for block in model.layers:            # shard each transformer block separately
    fully_shard(block, mp_policy=mp)
fully_shard(model, mp_policy=mp)      # root module last
```
Wrap per transformer block so only one block's full params are materialized at a time. Save with `torch.distributed.checkpoint` (DCP), not `torch.save(model.state_dict())` on rank 0.

## DeepSpeed ZeRO Config
```json
{
  "train_micro_batch_size_per_gpu": 4,
  "gradient_accumulation_steps": 8,
  "bf16": { "enabled": true },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "gradient_clipping": 1.0
}
```
`offload_optimizer` / `offload_param` (CPU or NVMe) trade large slowdowns for capacity; use only when GPUs are insufficient.

## Diagnostics
```bash
export NCCL_DEBUG=INFO          # prints chosen transport (NVLink/P2P, IB, Socket) and algorithms
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
```
- "NET/Socket" in NCCL logs on an InfiniBand/RoCE cluster means RDMA is not used: check drivers, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`.
- Benchmark the fabric with `nccl-tests` (`all_reduce_perf -b 8 -e 4G -f 2 -g 8`) before blaming the model.
- Track MFU (model FLOPs utilization): achieved FLOPs / peak FLOPs. Compare against your own single-node baseline.

## Anti-Patterns
- Forgetting `sampler.set_epoch()` (same order every epoch) or using a non-distributed sampler (every rank sees the same data).
- Rank-0-only `torch.save` of an FSDP model (gathers everything on one GPU, OOM).
- TP across nodes over Ethernet/InfiniBand without measuring; it is usually communication bound.
- Changing world size without rescaling learning rate / global batch.
- Hangs from rank-dependent control flow (one rank skips a collective).

## References
- PyTorch distributed docs (DDP, FSDP2, DCP), DeepSpeed ZeRO docs, ZeRO paper (Rajbhandari et al., 2020).
- Related: `../rdma-roce/SKILL.md` (fabric), `../flash-attention/SKILL.md` (activation memory).

Wersje

W bazie nie ma innych skilli o nazwie distributed-training.

Migawka bazy z 13.11.2026 · wersja paczki cb2f94a64ea3 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 95pytorch-lightning
    Upraszcza trening modeli PyTorch dzięki klasie Trainer, callbackom i rozproszonemu treningowi DDP, FSDP lub DeepSpeed.
  • 100alterlab-pytorch-lightning
    Organizuje trenowanie sieci neuronowych w PyTorch Lightning: moduły, Trainer dla wielu GPU i TPU, potoki danych, callbacki, logowanie do W&B oraz trenowanie rozproszone DDP, FSDP i DeepSpeed.
  • 90distributed-training-scaling
    Skaluje trening z jednego GPU na wiele przez tryby data parallel i sharded FSDP, nakłada komunikację na obliczenia i odczytuje krzywą wydajności, by znaleźć, gdzie skalowanie traci.
  • 100ml-training
    Trenuje, debuguje, skaluje i poprawia modele ML: pętle w PyTorch, TensorFlow, JAX i Hugging Face, trenowanie rozproszone DDP, FSDP i DeepSpeed, mieszana precyzja, przeuczenie i optymalizacja pamięci.
  • 90model-inference-optimize
    Optymalizuje wnioskowanie modeli: profilowanie, TensorRT, ONNX, PyTorch, FP8, cache i kontrolę jakości po zmianach.
  • 100distributed-training-setup
    Prawdopodobnie konfiguruje rozproszone trenowanie modeli uczenia maszynowego, na podstawie nazwy i kategorii ML Training.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.