Skip to content
Skills Library
DevOps & cloud · Analysis

coreweave-gpu-node-forensics

/coreweave-gpu-node-forensics

Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg / nvidia-smi blob, so a bad card does not silently kill a multi-day training run. Use when a GPU throws an Xid error, a node "fell off the bus", a training run stalls or NCCL hangs on one rank, or you need to know whether to replace, reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus", "coreweave gpu dead", "should I RMA this GPU", "gpu node triage".

Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
8
Scripts
1
HTML comments
0
URLs in SKILL.md
3
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/triage.py

What it does and when to use it

Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg / nvidia-smi blob, so a bad card does not silently kill a multi-day training run. Use when a GPU throws an Xid error, a node "fell off the bus", a training run stalls or NCCL hangs on one rank, or you need to know whether to replace, reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus", "coreweave gpu dead", "should I RMA this GPU", "gpu node triage".

Triggers from the description

fell off the busxid errorgpu fell off the buscoreweave gpu deadshould I RMA this GPUgpu node triage
Original author description

Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg / nvidia-smi blob, so a bad card does not silently kill a multi-day training run. Use when a GPU throws an Xid error, a node "fell off the bus", a training run stalls or NCCL hangs on one rank, or you need to know whether to replace, reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus", "coreweave gpu dead", "should I RMA this GPU", "gpu node triage".

How it works

  1. Overview
  2. Prerequisites
  3. Instructions
  4. Output
  5. Error Handling
  6. Examples
  7. Resources
Original SKILL.md headings · 234 lines, 1,800 words
SKILL.md preview
---
name: coreweave-gpu-node-forensics
description: |
  Triage a dead or degraded GPU on a CoreWeave node fast — decide reschedule
  vs GPU-reset vs node-reboot vs RMA from an Xid code or a pasted dmesg /
  nvidia-smi blob, so a bad card does not silently kill a multi-day training run.
  Use when a GPU throws an Xid error, a node "fell off the bus", a training run
  stalls or NCCL hangs on one rank, or you need to know whether to replace,
  reset, or just reschedule. Trigger with "xid error", "gpu fell off the bus",
  "coreweave gpu dead", "should I RMA this GPU", "gpu node triage".
allowed-tools: Read, Bash(python3:*), Bash(nvidia-smi -q:*), Bash(kubectl get:*), Bash(dmesg:*)
version: 1.11.0
author: Jeremy Longshore <jeremy@intentsolutions.io>
license: MIT
compatibility: Designed for Claude Code
tags: [saas, coreweave, gpu-cloud, reliability, xid]
---

# CoreWeave GPU Node Forensics

> **Community-contributed.** Not affiliated with, endorsed by, or sponsored by
> CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
> "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are
> cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one
grounded move — **reschedule, reset-gpu, reboot-node, rma, watch, or
app-bug-not-hardware** — from an NVIDIA Xid code or a pasted `dmesg` /
`nvidia-smi` blob. The classification is deterministic: a bundled script does the
mapping so the agent never guesses whether a card is dead, degraded, or fine.

## Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars
and days of wall-clock gone — because one rank stalls the whole collective. The
expensive mistakes are triage mistakes: RMAing a healthy card for an app bug,
restarting a job onto a GPU whose memory error was **uncontained**, or manually
uncordoning a node the lifecycle controller is trying to replace. This skill kills
that ambiguity.

The decision logic is grounded in the **NVIDIA Xid error catalog**
(<https://docs.nvidia.com/deploy/xid-errors/>) and CoreWeave's node-lifecycle /
cordon behavior. The math-of-the-matter — which Xid means what, and how
row-remapper state overrides it — lives in `scripts/triage.py` as a table the LLM
does not get to re-litigate. Deep domain knowledge (the full code→action table,
the row-remap decision, the cordon rules) loads from `references/` on demand.

**The headline is the Xid 94-vs-95 split.** A *contained* memory error (94) cost
you one job restart on a healthy node; an *uncontained* one (95) means the GPU
could not isolate the fault and everything it touched is suspect. Getting that one
bit wrong is the difference between a 30-second reschedule and a run that quietly
trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it **recommends** the cordon / drain /
reset / RMA next-step but its tools are scoped read-only (`nvidia-smi -q`,
`kubectl get`, `dmesg`) — it never runs a reset, a reboot, or an uncordon itself.

## Prerequisites

- **The failure evidence.** Either an Xid number, or a pasted `dmesg` /
  `nvidia-smi` dump. The skill works from a paste alone — no live cluster access
  required — which is the common case (an operator pastes what the run's logs
  showed).
- **Optional live access** for corroboration: `kubectl` context on the CoreWeave
  cluster (read-only is enough), and `nvidia-smi` on the node. If neither is
  available the skill still triages from the paste.
- **`python3`** to run the deterministic classifier (`scripts/triage.py`, stdlib
  only — no dependencies).

No secrets are handled. All commands are read-only queries.

## Instructions

The pipeline is **capture → classify → act**. The classifier is authoritative for
the verdict; `references/` supplies the "why" when a case needs depth.

### Step 1: Capture the evidence

If the user has not already pasted it, ask for (or read) the fault signal. The two
richest sources:

```bash
dmesg -T | grep -i xid                       # the Xid line(s) with timestamps
nvidia-smi -q -d ROW_REMAPPER,ECC,PERFORMANCE # remap state, ECC counts, throttle reasons
```

On a CoreWeave node you can also check who owns any cordon before acting:

```bash
kubectl get node NODE -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints}'
```

### Step 2: Run the deterministic triage

Feed the Xid code, or the whole blob, to the classifier. **Do not classify by
hand** — the script owns the Xid→action mapping and the row-remap override.

```bash
# From an Xid code:
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95

# With row-remapper state (Xid 63/64 or a DBE):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes

# From a pasted dmesg / nvidia-smi blob (file or stdin):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"
```

The script emits a `VERDICT` block plus a JSON object
(`{classification, severity, action, why, next_command, cordon_rule?, unverified?}`).
Add `--json` for machine-readable output only.

When a blob carries several Xids, the **most severe** one governs the move and the
rest are reported as `co_occurring_xids` — a co-occurring app-side Xid 43 next to a
hardware Xid 79 does not soften the "reboot the node" verdict.

### Step 3: Read the verdict and act on the action

Present the `action` and the `next_command` to the user in plain language. The six
actions and what each means live in
[`${CLAUDE_SKILL_DIR}/references/xid-triage-table.md`](references/xid-triage-table.md).
Load it when the user wants the full table or asks about an Xid not in the summary.

- `reschedule` — restart the failed rank; the node stays in service.
- `reset-gpu` — drain the GPU and reset it; re-run any job that shared it.
- `reboot-node` — the card is off the bus; only a bare-metal reboot returns it.
- `rma` — terminal hardware fault; the card must be replaced.
- `watch` — correctable / trending; monitor, do not act yet.
- `app-bug-not-hardware` — the app faulted, not the GPU. **Do NOT RMA.**

### Step 4: The 94-vs-95 headline (load-bearing — get this right)

If the Xid is **94 or 95**, state the containment explicitly, because the two look
almost identical in the logs and lead to opposite actions:

- **Xid 94 (CONTAINED)** → `reschedule`. The error was isolated to the app's
  context; the GPU and node are healthy. Restart the job. Do not reset or RMA.
- **Xid 95 (UNCONTAINED)** → `reset-gpu`. The GPU could not isolate it; every
  context it touched is suspect. Drain, reset, and re-run anything that shared the
  card — otherwise the run may continue on corrupt state.

### Step 5: Row-remap (Xid 63 / 64 / 48) — routine vs terminal

For any ECC/DBE or row-remap Xid, the `nvidia-smi -q -d ROW_REMAPPER` fields
override the base verdict. Pass them in (`--pending`, `--remap-failure`) and let
the script decide. The rule:

- `Remapping Failure Occurred: Yes` → **`rma`** (terminal — sparing failed).
- `Pending: Yes` (no failure) → **`reset-gpu`** (routine — applies on reset).

Full logic + how to read the histogram:
[`${CLAUDE_SKILL_DIR}/references/row-remap-decision.md`](references/row-remap-decision.md).

### Step 6: The cordon hard rule (never violate)

Whenever the action is a hardware move (`reset-gpu`, `reboot-node`, `rma`), the
verdict carries a `cordon_rule`. Surface it verbatim:

> **Never manually uncordon a CoreWeave health cordon** — the node-lifecycle
> controller owns cordon/uncordon and is driving remediation. Uncordoning
> re-admits the run onto known-bad hardware and races the controller.

Determine cordon provenance before touching schedulability. Full guidance:
[`${CLAUDE_SKILL_DIR}/references/cordon-rules.md`](references/cordon-rules.md).

## Output

- **A verdict** — the deterministic `{classification, severity, action, why,
  next_command}` for the governing signal, presented in plain language with the
  single next command an operator runs.
- **The 94-vs-95 call stated explicitly** when either fires — contained →
  reschedule, uncontained → reset — never conflated.
- **The cordon rule** attached to every hardware action, verbatim, so nobody
  uncordons a node the controller is replacing.
- **Co-occurring Xids** listed when a blob carried several, with the most-severe
  one named as governing.
- **`[unverified]` hedges** surfaced honestly — e.g. CoreWeave's exact auto-RMA
  thresholds and health-cordon taint strings are not publicly published, and the
  skill says so rather than inventing a number.

## Error Handling

| Situation | Cause | Response |
|-----------|-------|----------|
| Unmapped Xid | Xid not in the triage table | Script returns a conservative `reset-gpu` default, flags it `[unverified]`, and points at the NVIDIA catalog. Do not RMA on an unmapped Xid. |
| Xid 94 vs 95 ambiguity | Both are "ECC error" in casual logs | Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset). |
| Row-remap without an Xid | Only `nvidia-smi` pasted | Script reads `Remapping Failure Occurred` / `Pending` and decides from those alone. |
| App-side Xid mistaken for hardware | Xid 13/31/43/45 | Classed `app-bug-not-hardware`; the fix is compute-sanitizer on the job, **never** an RMA. |
| Multiple Xids in one blob | Cascade (e.g. 43 then 79) | Most-severe governs; others listed as `co_occurring_xids`. |
| Empty / no signal | Blob has no Xid, remap, or thermal line | Script returns `watch` and asks for fresh `dmesg` + `nvidia-smi -q` evidence. |
| Thermal throttle, not a fault | `HW/SW Thermal Slowdown : Active` | Classed `watch` — healthy but hot; investigate cooling, not the card. |
| Tempted to uncordon a stalled node | Health cordon owned by the controller | Refuse. Surface the cordon rule; let remediation run. |

## Examples

### Example 1: "Xid 94 on one rank — do I need to replace the GPU?"

```text
VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION:  reschedule
WHY:     The error was CONTAINED to the faulting application's context — the GPU
         and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT:    Restart or reschedule the failed rank; the node stays in service.
```

No replacement. The containment did its job — restart the rank and keep the node.

### Example 2: "Xid 95 — same run, different node"

```text
VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION:  reset-gpu
WHY:     The error was UNCONTAINED — the GPU could not isolate it, so every context
         it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT:    Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON:  Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
         controller owns cordon/uncordon. Let it drain and replace the node.
```

Opposite of Example 1 despite looking identical in the logs: drain, reset, and
re-run the shared work — do not just restart.

### Example 3: "GPU fell off the bus"

`python3 scripts/triage.py --xid 79` → `reboot-node`. The card is unreachable on
the PCIe bus and returns only after a bare-metal reboot; expect the controller to
cordon and reboot — let it, and RMA only if it recurs after the reboot.

### Example 4: "Xid 63 with a remap failure"

`python3 scripts/triage.py --xid 63 --remap-failure yes` → `rma`. The remapper
tried to swap in a spare row and physically could not — terminal. Cordon, drain,
open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is
`[unverified]`.)

### Example 5: "Xid 43 killed my job"

`python3 scripts/triage.py --xid 43` → `app-bug-not-hardware`. Software-induced
fault — the fix is in the application (run it under compute-sanitizer), not an RMA.
Pulling the card would waste healthy hardware and never fix the job.

## Resources

- [`${CLAUDE_SKILL_DIR}/references/xid-triage-table.md`](references/xid-triage-table.md) — the full Xid→action table, the six actions, the 94-vs-95 distinction, cited to the NVIDIA Xid catalog.
- [`${CLAUDE_SKILL_DIR}/references/row-remap-decision.md`](references/row-remap-decision.md) — the `ROW_REMAPPER` routine-vs-terminal decision and how to read the histogram.
- [`${CLAUDE_SKILL_DIR}/references/cordon-rules.md`](references/cordon-rules.md) — CoreWeave health-cordon ownership and the safe operator path per action.
- [NVIDIA Xid Errors reference](https://docs.nvidia.com/deploy/xid-errors/)
- [NVIDIA GPU Memory Error Management (row remapping)](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/index.html)
Files in this package (8)
  • ARD.md 8 KB
  • PRD.md 8 KB
  • SKILL.md 13 KB
  • eval-spec.yaml 7 KB
  • references/cordon-rules.md 4 KB
  • references/row-remap-decision.md 3 KB
  • references/xid-triage-table.md 5 KB
  • scripts/triage.py 26 KB

Versions

The database has no other skills named coreweave-gpu-node-forensics.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version c7c50afd9a9f (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100coreweave-core-workflow-b
    Run distributed GPU training jobs on CoreWeave with multi-node PyTorch. Use when training models across multiple GPUs, setting up distributed training, or running fine-tuning jobs on CoreWeave H100 clusters. Trigger with phrases like "coreweave training", "coreweave multi-gpu",…
  • 100coreweave-common-errors
    Diagnose and fix CoreWeave GPU scheduling, pod, and networking errors. Use when pods are stuck Pending, GPUs are not allocated, or experiencing CUDA and NCCL errors. Trigger with phrases like "coreweave error", "coreweave pod pending", "coreweave gpu not found", "coreweave debug", "fix coreweave".
  • 90nvidia-ai-operations-day2
    Use this skill when reviewing day-2 operations of NVIDIA GPU fleets — DCGM exporter and DCGM-Diag posture, GPU telemetry into Prometheus/Grafana, MIG partitioning lifecycle, GPU health and Xid error response, fleet upgrade paths, and incident response for GPU-failure modes.…
  • 100runai-get-cluster-health-summary
    Point-in-time Run:ai cluster and node-health check: a HEALTHY, DEGRADED, or STUCK verdict per node pool with the dominant blocker — NotReady nodes, offline or failing GPU capacity, GPU counts, allocation vs utilization, the pending backlog, and a stuck scheduler vs a full pool.…
  • 100gpu-accelerator-infrastructure-for-ml-training
    Guides standing up GPU accelerator infrastructure on Kubernetes for ML training and serving workloads — installing the NVIDIA GPU Operator, partitioning GPUs with Multi-Instance GPU (MIG), designing GPU node pools, and scheduling/bin-packing training and serving pods onto that…
  • 100tao-run-on-kubernetes
    Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod for one node, Indexed Jobs for multi-node distributed training. Use when running on EKS / GKE / AKS / on-prem clusters with the NVIDIA GPU Operator installed, or when…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.