Skip to content
Skills Library
AI & agents · Analysis

debug

/debug

Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.

Repository
phd-skills ↗
Licence
MIT
Last repository change
11.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 11.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.

Original author description

Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.

How it works

  1. When to run
  2. Five-step protocol
  3. What to avoid
  4. Output
Original SKILL.md headings · 122 lines, 774 words
SKILL.md preview
---
name: debug
description: Evidence-before-action diagnosis of failing ML experiments. Probes the system before guessing causes, process list, dmesg, GPU stats, log scrollback, checkpoint state, then states a hypothesis as a hypothesis and runs a smoke before claiming a root cause. Use when the user asks why a run is failing, diverging, OOMing, hanging, slow, producing weird metrics, has crashed, or asks to debug, diagnose, troubleshoot, or investigate a training issue.
---

# Debug: evidence-before-action investigation

The most expensive class of mistake in ML debugging is asserting a cause based on plausibility, then attempting a "fix" that masks the real problem. This skill enforces the discipline of probe → hypothesis → smoke → controls → claim, in that order.

The agentic Stop hook routes here from `reason` when an assistant claims a cause without backing tool output.

## When to run

The user just said any of:

- "why is X failing / diverging / NaN / OOM / hung / slow / crashed"
- "the loss is going up", "metrics look weird", "GPU util is 0"
- "debug this", "diagnose", "troubleshoot", "investigate this run"
- pasted a log excerpt asking what's wrong

## Five-step protocol

### Step 1: cheap probes

Before forming any hypothesis, gather the cheap evidence. None of these cost more than a few seconds:

**Process state:**

```bash
ps aux | grep -E '(python|train|torchrun|accelerate)' | grep -v grep
```

Is the process still running? Zombie? Defunct? Multiple instances?

**Kernel / system events:**

```bash
dmesg | tail -100 # OOM kills, hardware errors, NFS errors
journalctl -xe --since "1 hour ago" | tail -50
```

**GPU state:**

```bash
nvidia-smi
nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv
```

Is the GPU even being used? Idle GPU during "training" means the process is blocked on data loading or has died.

**Disk / filesystem:**

```bash
df -h /path/to/run-dir
du -sh /path/to/run-dir/*
```

Out of disk? Checkpoints not being written?

**Log scrollback:**
Read the last few hundred lines of the training log. Don't trust the user's summary, they may have skimmed. Look for:

- exception tracebacks
- repeated "loss=NaN" or "grad_norm=Inf"
- early-stop announcements (the run may have completed normally)
- the _last_ successful epoch / step (where did progress stop)

**Checkpoint state:**

```bash
ls -la /path/to/run-dir/checkpoints/
```

When was the last checkpoint written? What does its size suggest? An empty `.pt` is different from a 2GB one cut short.

### Step 2: hypothesis (labeled as hypothesis)

After the probe, state what _might_ be happening, explicitly framed as a hypothesis:

> "Hypothesis: the run is OOMing because dmesg shows `oom-kill` 3 minutes ago and the process is gone. Alternative hypotheses I haven't ruled out: (a) NFS write timeout, (b) explicit kill from a sibling process."

Never skip to "the cause is X." The hypothesis labels what you don't yet know.

### Step 3: smoke run

The cheapest way to confirm or refute a hypothesis is to reproduce the failure shape under a controlled condition:

- **OOM hypothesis**: rerun with `batch_size=1` for 1 step. If it survives, OOM is confirmed; if it fails the same way, OOM is wrong.
- **Data hypothesis**: rerun with a synthetic in-memory dataset. If it works, the data path is implicated.
- **Model hypothesis**: forward pass only on a single batch with `eval()` mode. Loss finite? Outputs sane?
- **Optimizer hypothesis**: rerun with `lr=0`. If the loss still explodes, the loss itself is broken (not the optimizer).
- **Distributed hypothesis**: rerun on 1 GPU. If it works, DDP / NCCL is implicated.

A 30-second smoke beats a 30-minute restart-and-pray.

### Step 4: controls

If the smoke is ambiguous, run a control: change exactly one variable from the failing config and rerun the smoke. The differences narrow what mechanism is responsible.

Common control axes (change one at a time):

- single-source vs multi-source data
- default workers vs adjusted workers
- mixed-precision on vs off
- gradient checkpointing on vs off
- torch.compile on vs off

### Step 5: claim cause

Only after evidence stacks up, probe, smoke, control, do you assert a cause. The claim should cite the specific tool output that proves it:

> "Root cause: NFS write timeout. Evidence: dmesg shows `nfs server X not responding` at 14:23 (the same minute the last checkpoint was written), and the smoke with `batch=1` reproduces the timeout. Recommended fix: bind-mount a local scratch dir for checkpoints and rsync to NFS at end of epoch."

If the evidence isn't stacking up, do not promote a hypothesis to a cause. Say "I don't yet know" and propose the next probe.

## What to avoid

- "It's probably X, let me try Y" → no. Probe first.
- Restarting the run with a small change as the diagnostic. Smoke first, then restart deliberately.
- Citing only the user's narrative as evidence: re-read the actual log.
- Stopping at the first plausible cause when artifacts contradict it.

## Output

A concise diagnostic report: (1) what the probes showed, (2) the hypothesis, (3) the smoke outcome, (4) the cause-or-uncertain verdict, (5) the recommended next action. Each claim cites the tool output that backs it.

Versions

RepositoryScoreLicence
This page · fcakyon/phd-skills100MIT
ericrisco/rsc-harness100MIT
nicsuzor/academicops100MIT
duronext/jig100MIT
drosam/agents100MIT
rizukirr/vibekit100MIT
DingyangLyu/MatClaw100Apache-2.0
Kanevry/session-orchestrator100MIT
kdlbs/kandev100AGPL-3.0

Database snapshot: 13.11.2026 · package version 1678733dd034 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100why-slow
    In-session diagnostic for "why is Claude hanging / slow right now?" Reads recent startup-timing log entries, current MCP process list, latest sync-pull debug log, and worktree count, then prints a focused top-suspect report. Use when a session feels stuck or a tool call is…
  • 100hypothesis-driven-debugging
    Investigate build failures, test errors, runtime crashes, or unexpected behavior through systematic minimal reproduction, multi-hypothesis testing, and evidence-based verification. Use when asked to \"debug\", \"investigate a failure\", \"find root cause\", \"troubleshoot\",…
  • 100eval-triage
    Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.
  • 100output-debug-workflow
    Debug Output SDK workflow issues. Use when user reports a workflow failing, erroring, hanging, producing wrong results, or asks to debug, troubleshoot, or investigate a workflow execution.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.