mcore-testing
/mcore-testing
Opisuje system testów Megatron-LM: układ testów, receptury YAML, dodawanie i uruchamianie testów jednostkowych i funkcjonalnych, wartości wzorcowe, filtry markerów i zgodność z CI.
- Autor
- nvidia ↗
- Repozytorium
- megatron-lm ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 27.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 5
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Opisuje system testów Megatron-LM: układ testów, receptury YAML, dodawanie i uruchamianie testów jednostkowych i funkcjonalnych, wartości wzorcowe, filtry markerów i zgodność z CI.
Wyzwalacze z opisu
Oryginalny opis (EN)
Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.
Jak działa
- Answer-First Testing Facts
- Test Layout
- How Tests Execute
- Recipe YAML Structure
- Running Unit Tests Locally
- Adding a Unit Test
- Adding a Functional / Integration Test
- Common Pitfalls
Podgląd SKILL.md
---
name: mcore-testing
description: Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.
license: Apache-2.0
metadata:
author: Oliver Koenig <okoenig@nvidia.com>
---
# Testing Guide
---
## Answer-First Testing Facts
For questions about disabling tests without deleting them:
- Functional recipe entries stay in YAML; disable by suffixing scope with
`-broken`, for example `scope: [mr-github]` -> `scope: [mr-github-broken]`.
- Unit-test skips use pytest markers instead: `@pytest.mark.flaky_in_dev` skips
in the default dev environment, and `@pytest.mark.flaky` skips in LTS.
- Do not delete the test case or recipe entry when the goal is discoverability
and easy re-enable.
---
## Test Layout
```text
tests/
├── unit_tests/ # pytest, 1 node × 8 GPUs, torch.distributed runner
├── functional_tests/ # end-to-end shell + training scripts
│ └── test_cases/
│ └── {model}/{test_case}/
│ ├── model_config.yaml # training args
│ └── golden_values_{env}_{platform}.json
└── test_utils/
├── recipes/
│ ├── h100/ # YAML recipes for H100 jobs
│ └── gb200/ # YAML recipes for GB200 jobs
└── python_scripts/ # helpers (recipe_parser, golden-value download, …)
```
---
## How Tests Execute
The GitHub Actions runner invokes `launch_nemo_run_workload.py`, which uses
**nemo-run** to launch a `DockerExecutor` container. The repo is bind-mounted
at `/opt/megatron-lm`; training data is mounted at `/mnt/artifacts`.
**Unit tests** are dispatched through `torch.distributed.run`:
- Ranks 0 and 3 are tee-d to stdout; all other ranks write only to log files.
- Per-rank log files land at `{assets_dir}/logs/1/` and are uploaded as a
GitHub artifact after the run.
**Functional tests** are driven by
`tests/functional_tests/shell_test_utils/run_ci_test.sh`. Only rank 0 runs the
pytest validation step; training output from all ranks is uploaded as an artifact.
**Flaky-failure auto-retry**: `launch_nemo_run_workload.py` retries up to
**3 times** for known transient patterns (NCCL timeout, ECC error, segfault,
HuggingFace connectivity, …) before declaring a genuine failure.
---
## Recipe YAML Structure
Recipes live in `tests/test_utils/recipes/` and are parsed by
`tests/test_utils/python_scripts/recipe_parser.py`. Each file expands a
cartesian `products` block into individual workload specs:
```yaml
type: basic
format_version: 1
maintainers: [mcore]
loggers: [stdout]
spec:
name: "{test_case}_{environment}_{platforms}"
model: gpt # maps to tests/functional_tests/test_cases/{model}/
build: mcore-pyt-{environment}
nodes: 1
gpus: 8
n_repeat: 5
platforms: dgx_h100
time_limit: 1800
script_setup: |
...
script: |-
bash tests/functional_tests/shell_test_utils/run_ci_test.sh ...
products:
- test_case: [my_test]
products:
- environment: [dev, lts]
scope: [mr-github]
platforms: [dgx_h100]
```
Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`,
`{environment}`, `{platforms}`, `{n_repeat}`.
### Disabling a Test Without Deleting It
To temporarily disable a test case in a recipe YAML, suffix its `scope` value
with `-broken` — **do not delete the entry**:
```yaml
# before (test runs in CI)
scope: [mr-github]
# after (test is skipped; entry preserved for easy re-enable)
scope: [mr-github-broken]
```
---
## Running Unit Tests Locally
All unit tests initialize a `torch.distributed` group, so every invocation
requires GPU access and must go through `torch.distributed.run`:
```bash
# Full suite
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests
# Single file
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests/models/test_gpt_model.py
# Single test
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests/models/test_gpt_model.py::TestGPTModel::test_constructor
# Filter by name substring
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests -k optimizer
```
### Marker filters
```bash
# Exclude flaky tests during development
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests -m "not flaky and not flaky_in_dev"
# Include experimental tests
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
tests/unit_tests --experimental
```
### CI parity
Use `tests/unit_tests/run_ci_test.sh` to reproduce a CI bucket failure exactly.
For ad-hoc runs, prefer the direct `torch.distributed.run` invocations above.
### Gotchas
- `pyproject.toml` sets `addopts = --durations=15 -s -rA` — stdout is not
captured (`-s`), so ranks interleave during multi-rank runs. Override with
`--capture=fd` when debugging a specific rank.
- `tests/unit_tests/conftest.py` looks for test data under `/opt/data` and
attempts a download if missing. Supply it manually or skip data-dependent
tests when running outside the canonical container.
---
## Adding a Unit Test
1. Create `tests/unit_tests/<category>/test_<name>.py`.
2. Use fixtures from `tests/unit_tests/conftest.py`.
3. Apply markers as needed:
- `@pytest.mark.internal` — skipped on `legacy` tag
- `@pytest.mark.flaky_in_dev` — skipped in `dev` environment (CI default; use this to disable a flaky test without blocking the standard pipeline)
- `@pytest.mark.flaky` — skipped in `lts` environment
- `@pytest.mark.experimental` — `latest` tag only
4. Verify locally (see Running Unit Tests Locally above).
5. If the test needs a dedicated CI bucket, add an entry to
`tests/test_utils/recipes/h100/unit-tests.yaml`.
6. If the change adds or modifies a GPU kernel (Triton, `jit_fuser` /
`torch.compile`, CUDA extension, TE or external-library dispatch, or a
scatter/index accumulation), add or update its bit-exact replay test under
`tests/unit_tests/determinism/kernels/` and register it in
`tests/unit_tests/determinism/kernels/manifest.py`. The `linting` CI job
(`tools/check_kernel_determinism_coverage.py`) fails kernel PRs without
this; the `determinism-exempt` label overrides it for non-numeric edits.
See `docs/developer/determinism/testing.md`.
---
## Adding a Functional / Integration Test
1. Create `tests/functional_tests/test_cases/<model>/<test_name>/`.
2. Write `model_config.yaml` with `MODEL_ARGS`, `ENV_VARS`, and `TEST_TYPE`.
3. Add a YAML recipe under `tests/test_utils/recipes/h100/` (and `gb200/` if
needed). Required fields: `scope`, `environment`, `platform`, `n_repeat`,
`time_limit`.
4. Push the PR, add the label **"Run functional tests"** to trigger a full run.
5. After a successful run, download golden values:
```bash
python tests/test_utils/python_scripts/download_golden_values.py \
--source github --pipeline-id <run-id>
```
6. Commit the downloaded golden values.
Golden values keep the full `float32` precision of the TensorBoard scalars,
record it as `"value_precision": "full"`, and deterministic test cases are
compared bit-exactly against them. Never round or hand-edit them:
`tools/check_golden_values.py` (run by the `linting` CI job on changed golden
files) rejects golden files of deterministically compared cases whose metrics
lack the `full` marker. Legacy files (no marker) are still compared at five
decimals until they are regenerated from a CI run.
---
Golden comparisons report absolute and relative errors; see [diagnostics](../../docs/developer/determinism/testing.md).
## Common Pitfalls
| Problem | Cause | Fix |
|---------|-------|-----|
| Test passes locally but fails in CI | Different environment or data path | Check `DATA_PATH`, `DATA_CACHE_PATH`, and the `environment` tag (`dev` vs `lts`) |
| Golden value mismatch after a code change | Numerical regression | Download new golden values via `download_golden_values.py` after a clean run |
| `cicd-integration-tests-gb200` not triggered | GB200 jobs require maintainer status | Ask a maintainer to trigger, or add the `Run functional tests` label |
Pliki w paczce (5)
- BENCHMARK.md 3 KB
- SKILL.md 8 KB
- skill-card.md 2 KB
- skill.oms.sig 4 KB
- evals/evals.json 1 KB
Wersje
W bazie nie ma innych skilli o nazwie mcore-testing.
Migawka bazy z 13.11.2026 · wersja paczki e8c02cdd9c94 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100nvidia-megatron-core-update-golden-values
Odświeża wartości wzorcowe z uruchomienia workflow GitHub Actions, ocenia zmianę średnimi znormalizowanymi różnicami względnymi i przygotowuje podsumowanie do opisu pull requesta. - 100unit-tests
Pilnuje zasad dodawania testów jednostkowych w bklit-ui, żeby nie testować ponad miarę. - 100add-unit-tests
Prowadzi przez dodawanie testów jednostkowych do AReaL dla nowej funkcjonalności lub w celu zwiększenia pokrycia testami. - 100add-unit-tests
Prowadzi przez dodawanie testów jednostkowych do AReaL dla nowej funkcjonalności lub w celu zwiększenia pokrycia testami. - 100testing
Podaje dobre praktyki testów Dart, testów widgetów Fluttera i testów golden: nazwy, setUp, mocki mocktail i pomocnik pumpApp. - 100designing-tests
Projektuje i wdraża strategie testowania: dodawanie testów jednostkowych, integracyjnych i E2E, poprawę pokrycia, konfigurację infrastruktury testowej i diagnozę błędów testów.