Skip to content
Skills Library
Programming · Testing

mcore-testing

/mcore-testing

Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.

Repository
megatron-lm ↗
Licence
MIT
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
5
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.

Triggers from the description

how do I add a testrun unit testspytest failstest layoutgolden valuesrecipe YAMLmarker filter
Original author description

Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.

How it works

  1. Answer-First Testing Facts
  2. Test Layout
  3. How Tests Execute
  4. Recipe YAML Structure
  5. Running Unit Tests Locally
  6. Adding a Unit Test
  7. Adding a Functional / Integration Test
  8. Common Pitfalls
Original SKILL.md headings · 220 lines, 997 words
SKILL.md preview
---
name: mcore-testing
description: Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity. Use when adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.
license: Apache-2.0
metadata:
  author: Oliver Koenig <okoenig@nvidia.com>
---

# Testing Guide

---

## Answer-First Testing Facts

For questions about disabling tests without deleting them:

- Functional recipe entries stay in YAML; disable by suffixing scope with
  `-broken`, for example `scope: [mr-github]` -> `scope: [mr-github-broken]`.
- Unit-test skips use pytest markers instead: `@pytest.mark.flaky_in_dev` skips
  in the default dev environment, and `@pytest.mark.flaky` skips in LTS.
- Do not delete the test case or recipe entry when the goal is discoverability
  and easy re-enable.

---

## Test Layout

```text
tests/
├── unit_tests/          # pytest, 1 node × 8 GPUs, torch.distributed runner
├── functional_tests/    # end-to-end shell + training scripts
│   └── test_cases/
│       └── {model}/{test_case}/
│           ├── model_config.yaml          # training args
│           └── golden_values_{env}_{platform}.json
└── test_utils/
    ├── recipes/
    │   ├── h100/        # YAML recipes for H100 jobs
    │   └── gb200/       # YAML recipes for GB200 jobs
    └── python_scripts/  # helpers (recipe_parser, golden-value download, …)
```

---

## How Tests Execute

The GitHub Actions runner invokes `launch_nemo_run_workload.py`, which uses
**nemo-run** to launch a `DockerExecutor` container. The repo is bind-mounted
at `/opt/megatron-lm`; training data is mounted at `/mnt/artifacts`.

**Unit tests** are dispatched through `torch.distributed.run`:

- Ranks 0 and 3 are tee-d to stdout; all other ranks write only to log files.
- Per-rank log files land at `{assets_dir}/logs/1/` and are uploaded as a
  GitHub artifact after the run.

**Functional tests** are driven by
`tests/functional_tests/shell_test_utils/run_ci_test.sh`. Only rank 0 runs the
pytest validation step; training output from all ranks is uploaded as an artifact.

**Flaky-failure auto-retry**: `launch_nemo_run_workload.py` retries up to
**3 times** for known transient patterns (NCCL timeout, ECC error, segfault,
HuggingFace connectivity, …) before declaring a genuine failure.

---

## Recipe YAML Structure

Recipes live in `tests/test_utils/recipes/` and are parsed by
`tests/test_utils/python_scripts/recipe_parser.py`. Each file expands a
cartesian `products` block into individual workload specs:

```yaml
type: basic
format_version: 1
maintainers: [mcore]
loggers: [stdout]
spec:
  name: "{test_case}_{environment}_{platforms}"
  model: gpt              # maps to tests/functional_tests/test_cases/{model}/
  build: mcore-pyt-{environment}
  nodes: 1
  gpus: 8
  n_repeat: 5
  platforms: dgx_h100
  time_limit: 1800
  script_setup: |
    ...
  script: |-
    bash tests/functional_tests/shell_test_utils/run_ci_test.sh ...
products:
  - test_case: [my_test]
    products:
      - environment: [dev, lts]
        scope: [mr-github]
        platforms: [dgx_h100]
```

Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`,
`{environment}`, `{platforms}`, `{n_repeat}`.

### Disabling a Test Without Deleting It

To temporarily disable a test case in a recipe YAML, suffix its `scope` value
with `-broken` — **do not delete the entry**:

```yaml
# before (test runs in CI)
scope: [mr-github]

# after (test is skipped; entry preserved for easy re-enable)
scope: [mr-github-broken]
```

---

## Running Unit Tests Locally

All unit tests initialize a `torch.distributed` group, so every invocation
requires GPU access and must go through `torch.distributed.run`:

```bash
# Full suite
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests

# Single file
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests/models/test_gpt_model.py

# Single test
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests/models/test_gpt_model.py::TestGPTModel::test_constructor

# Filter by name substring
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests -k optimizer
```

### Marker filters

```bash
# Exclude flaky tests during development
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests -m "not flaky and not flaky_in_dev"

# Include experimental tests
uv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \
  tests/unit_tests --experimental
```

### CI parity

Use `tests/unit_tests/run_ci_test.sh` to reproduce a CI bucket failure exactly.
For ad-hoc runs, prefer the direct `torch.distributed.run` invocations above.

### Gotchas

- `pyproject.toml` sets `addopts = --durations=15 -s -rA` — stdout is not
  captured (`-s`), so ranks interleave during multi-rank runs. Override with
  `--capture=fd` when debugging a specific rank.
- `tests/unit_tests/conftest.py` looks for test data under `/opt/data` and
  attempts a download if missing. Supply it manually or skip data-dependent
  tests when running outside the canonical container.

---

## Adding a Unit Test

1. Create `tests/unit_tests/<category>/test_<name>.py`.
2. Use fixtures from `tests/unit_tests/conftest.py`.
3. Apply markers as needed:
   - `@pytest.mark.internal` — skipped on `legacy` tag
   - `@pytest.mark.flaky_in_dev` — skipped in `dev` environment (CI default; use this to disable a flaky test without blocking the standard pipeline)
   - `@pytest.mark.flaky` — skipped in `lts` environment
   - `@pytest.mark.experimental` — `latest` tag only
4. Verify locally (see Running Unit Tests Locally above).
5. If the test needs a dedicated CI bucket, add an entry to
   `tests/test_utils/recipes/h100/unit-tests.yaml`.
6. If the change adds or modifies a GPU kernel (Triton, `jit_fuser` /
   `torch.compile`, CUDA extension, TE or external-library dispatch, or a
   scatter/index accumulation), add or update its bit-exact replay test under
   `tests/unit_tests/determinism/kernels/` and register it in
   `tests/unit_tests/determinism/kernels/manifest.py`. The `linting` CI job
   (`tools/check_kernel_determinism_coverage.py`) fails kernel PRs without
   this; the `determinism-exempt` label overrides it for non-numeric edits.
   See `docs/developer/determinism/testing.md`.

---

## Adding a Functional / Integration Test

1. Create `tests/functional_tests/test_cases/<model>/<test_name>/`.
2. Write `model_config.yaml` with `MODEL_ARGS`, `ENV_VARS`, and `TEST_TYPE`.
3. Add a YAML recipe under `tests/test_utils/recipes/h100/` (and `gb200/` if
   needed). Required fields: `scope`, `environment`, `platform`, `n_repeat`,
   `time_limit`.
4. Push the PR, add the label **"Run functional tests"** to trigger a full run.
5. After a successful run, download golden values:

   ```bash
   python tests/test_utils/python_scripts/download_golden_values.py \
     --source github --pipeline-id <run-id>
   ```

6. Commit the downloaded golden values.

Golden values keep the full `float32` precision of the TensorBoard scalars,
record it as `"value_precision": "full"`, and deterministic test cases are
compared bit-exactly against them. Never round or hand-edit them:
`tools/check_golden_values.py` (run by the `linting` CI job on changed golden
files) rejects golden files of deterministically compared cases whose metrics
lack the `full` marker. Legacy files (no marker) are still compared at five
decimals until they are regenerated from a CI run.

---

Golden comparisons report absolute and relative errors; see [diagnostics](../../docs/developer/determinism/testing.md).

## Common Pitfalls

| Problem | Cause | Fix |
|---------|-------|-----|
| Test passes locally but fails in CI | Different environment or data path | Check `DATA_PATH`, `DATA_CACHE_PATH`, and the `environment` tag (`dev` vs `lts`) |
| Golden value mismatch after a code change | Numerical regression | Download new golden values via `download_golden_values.py` after a clean run |
| `cicd-integration-tests-gb200` not triggered | GB200 jobs require maintainer status | Ask a maintainer to trigger, or add the `Run functional tests` label |
Files in this package (5)
  • BENCHMARK.md 3 KB
  • SKILL.md 8 KB
  • skill-card.md 2 KB
  • skill.oms.sig 4 KB
  • evals/evals.json 1 KB

Versions

The database has no other skills named mcore-testing.

Database snapshot: 13.11.2026 · package version e8c02cdd9c94 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100nvidia-megatron-core-update-golden-values
    Refresh golden values from a GitHub Actions workflow run (failing-only or all jobs), score the change with average normalized relative differences, and produce a PR-ready summary. Use when the user asks to update goldens for a CI run, refresh golden values from a workflow ID, or…
  • 100unit-tests
    Guardrails for adding unit tests in bklit-ui without over-testing. Use when the user mentions unit test, unit tests, tests, test coverage, add tests, write tests, vitest, jest, or asks whether something should be tested.
  • 100add-unit-tests
    Guide for adding unit tests to AReaL. Use when user wants to add tests for new functionality or increase test coverage.
  • 100add-unit-tests
    Guide for adding unit tests to AReaL. Use when user wants to add tests for new functionality or increase test coverage.
  • 100testing
    Best practices for Dart unit tests, Flutter widget tests, and golden file tests, covering group and test naming, setUp lifecycle and isolation, mocking with package:mocktail, and the shared pumpApp helper. Use when writing, modifying, or reviewing tests, goldens included, that…
  • 100designing-tests
    Designs and implements testing strategies for any codebase. Use when adding tests, improving coverage, setting up testing infrastructure, debugging test failures, or when asked about unit tests, integration tests, or E2E testing.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.