implement-paper

---
name: implement-paper
description: Implement and run a paper reproduction from its spec (spec/<paper_id>.md) so that the reproducibility tests in tests/<paper_id>/ can pass. Builds on the paper's official GitHub code whenever possible (git submodule pinned at the spec's commit, with our changes as patch files), sets up a per-paper environment, runs smoke tests and released-checkpoint evaluations, estimates compute and asks before full training runs, and writes results/<paper_id>/*.json. If the spec is missing details it looks them up in the official code or the original paper and proposes spec updates, but only edits the spec after the user approves. Use this whenever the user wants to implement, re-implement, code up, set up, port, run or continue a paper reproduction, get the official code of a paper working, or make a paper's reproducibility tests pass, even if they only name the paper or its paper_id.
---

# Implement a paper from its spec

Input: a `paper_id` (or a paper the user names, mapped to its `spec/<paper_id>.md`).
Output: working code in `papers/<paper_id>/`, results in
`results/<paper_id>/<experiment_id>.json`, and an up-to-date
`papers/<paper_id>/REPRODUCTION_LOG.md`, with `pytest tests/<paper_id>` as the
judge.

The spec is the contract. It was written from the paper, and the tests are
derived from it. Your job is to make the code meet the spec faithfully, not to
make the tests green. A reproduction that honestly fails is useful; one that
quietly changed the method, the protocol or the targets is worse than none.

## 0. Preconditions

- `spec/<paper_id>.md` and `tests/<paper_id>/test_<paper_id>.py` exist. If not,
  stop and offer the `paper-to-specs-and-tests` skill first.
- Read the **whole spec** and the test file before writing code. Pay most
  attention to: §2 (implementation decision and fallback trigger), §4 (setup),
  §5 (evaluation protocol), §6 (targets), §7 (results contract), §8
  (assumptions).
- If `papers/<paper_id>/REPRODUCTION_LOG.md` already exists, this is a
  continuation: read it, pick up from its Status table and open spec questions,
  and don't redo finished steps.

## 1. Official code first

Use the official implementation whenever it is usable. It contains the
details papers leave out, and reproductions built on it are both more faithful
and cheaper. Concretely, by the spec's §2 decision:

- **Base on official repo** → bring it in as a submodule and get it running
  (porting it to a current stack if needed). Porting working code beats
  rewriting it.
- **Official repo as reference only** → write own code in `src/`, but still add
  the submodule (or keep a clone in the scratchpad) and read the official code
  for every detail the spec leaves open. Cite `official/<file>:<line>` in
  comments where you follow it.
- **Own implementation** → first spend a minute checking whether official code
  has appeared since the spec was written (the paper's page, the authors'
  GitHub, `huggingface.co/papers/<arxiv_id>`). If it has, that is a spec
  update: propose switching (step 3). Otherwise implement in `src/`.

Changing strategy (e.g. the spec's fallback trigger fires and you want to drop
the official code) is a spec change too. Gather the evidence, then ask.

Read `references/official-repo-workflow.md` before touching the official code.
It covers the submodule + patches workflow, the per-paper environment, porting
old code, and reading third-party code safely.

## 2. Scaffold the workspace

```bash
python .claude/skills/implement-paper/scripts/scaffold_paper.py <paper_id> \
    [--official-url <github url>] [--sha <commit>] [--python <version>]
```

It creates `papers/<paper_id>/` with `official/` (submodule at the spec's
pinned SHA, read from the spec header when `--sha` is omitted), `patches/`,
`src/`, `configs/`, `tests/`, `scripts/{setup_env,apply_patches,save_patch}.sh`
and `REPRODUCTION_LOG.md`. Then edit `scripts/setup_env.sh` to install what the
code needs and run it. The `.venv` lives in `papers/<paper_id>/.venv`, so each
paper gets the library versions its code needs.

Layout rules that keep reproductions reviewable:
- Changes to upstream code exist **only** as `patches/NNNN-<name>.patch`
  (create them with `scripts/save_patch.sh <name>`). One concern per patch.
- Own code, wrappers and entry points go in `src/`, configs in `configs/`
  (one per `experiment_id`), and unit tests of own code in `papers/<paper_id>/tests/`.
- Data, checkpoints and run outputs go outside `papers/` (`data/`,
  `checkpoints/`, `runs/`, all gitignored), never inside `official/`.

## 3. When the spec is not enough: propose, then ask

You will hit details the spec doesn't settle: a missing hyperparameter, an
ambiguous preprocessing step, an official config that disagrees with the
paper, a target that looks mis-copied. Resolve them in this order:

1. **Look it up.** Check the official code first (configs, defaults, training
   script), then the paper itself. Fetch it again if needed, following the
   `paper-to-specs-and-tests` skill's `references/fetching.md` (published
   version first, the arXiv LaTeX source for tables). Note exactly where the
   answer is.
2. **Write the proposal**: the spec section, the current text, the proposed
   text, the source, and what it affects (code only, or targets/tests too).
3. **Ask the user before editing the spec.** Show the proposal as a short
   diff in your message, then ask with AskUserQuestion. Options: *Apply*,
   *Apply as an assumption* (goes in §8 with its uncertainty), *Keep the spec
   as is*. Group related questions, up to 4 per call. Don't ask one at a time
   if several are pending.
4. **Until the user answers**, don't block on it when you can avoid it. Use
   the proposed value provisionally if the work is cheap to redo (it shows up
   as `provisional` in the log's Spec questions table), and continue with
   other parts. Block only when a wrong guess would waste real compute: never
   launch a full run on a provisional value.
5. **After approval**, edit the spec, append a line to its `## Change log`
   section (create it at the end if missing: date, section, change, source,
   "approved by user"), and mark the question `applied` in the log. If the
   change touches §6 (targets, tolerances, claims), update `TARGETS`/`CLAIMS` in
   the test file to match and run
   `python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <paper_id>`.

Why so strict: the spec and tests are the independent yardstick. If the
implementer can change them freely, the yardstick moves toward whatever the
code produces, and a passing test no longer means anything. The user has to
own every change to what "reproduced" means.

Spec changes covered by this rule include target values, tolerances,
experiments, the protocol, the implementation strategy, the pinned commit and
filled gaps. Don't ask about choices that are purely about code (file layout,
refactors, logging), and don't put them in the spec.

## 4. Implement faithfully

- **Follow the spec, not your taste.** Don't fix what looks like a flaw in the
  method, swap in a "better" optimizer, or modernize the architecture. If the
  official code and the paper disagree, that is a spec question (step 3).
- **Own code mirrors the paper**: name things as the paper does, reference
  equations (`# Eq. (3)`) and sections in comments, and keep the method's
  structure visible. Add focused unit tests in `papers/<paper_id>/tests/` for
  the parts that are easy to get subtly wrong: loss terms on hand-computed
  inputs, masking, shapes, schedules. When official code exists, compare your
  module's output against it on the same small input.
- **Results**: every experiment ends by writing its result file with
  ```bash
  python .claude/skills/implement-paper/scripts/write_results.py --paper-id <paper_id> \
      --experiment <experiment_id> --per-seed <metric>=v1,v2,v3 \
      --implementation official|official-modified|own --config <path> --run-dir <dir> --notes "<deviations>"
  ```
  It records provenance (bench commit, official commit, patch hash). Use the
  metric keys and units of spec §7 exactly. A unit mismatch (fraction vs %) is
  the most common way to fail a correct reproduction.
- **Log as you go**: every patch and deviation goes in the log's Deviations
  table, every spec question in Spec questions, and environment facts in
  Environment.

## 5. Run: cheap checks yourself, full runs after asking

Read `references/running-experiments.md`. In short:

1. **Smoke run** of each experiment pipeline on tiny data, written under a
   `_smoke` experiment ID. Measure throughput here.
2. **Stage 0**: when checkpoints are released, evaluate them with the official
   eval code and run the tests. If they don't match the paper, fix the
   eval/data pipeline before any training.
3. **Compute estimate** per experiment from measured throughput. Show the user
   a table and **ask which full runs to launch** (AskUserQuestion). Training
   jobs hold shared GPUs for hours or days, so that is the user's call.
4. **Full runs** on idle local GPUs (check `nvidia-smi`), detached with
   `nohup`, one run dir per run under `runs/<paper_id>/<experiment_id>/`,
   each recorded in the log.
5. After each run: evaluate on the spec's protocol, write results, then run
   `python -m pytest tests/<paper_id> -v`.

A failing test is information, not something to optimize away. Never widen a
tolerance, select checkpoints or hyperparameters on the test set, drop bad
seeds, or rerun until a seed lands in range. Look for the cause; if none is
found, report the miss with what you checked.

## 6. Report

End each session with a short report to the user, and update the log's Status
table to match:
- per experiment: stage reached, and local value vs paper value ± tolerance
  (from the pytest output)
- the implementation route actually used, and the patches and deviations that
  can affect numbers
- spec questions: applied, pending (what you need from the user), provisional
- what is still to run, with the compute estimate, and anything blocked

## Files in this skill

- `scripts/scaffold_paper.py`: creates `papers/<paper_id>/`, adds the official
  submodule at the pinned SHA, and writes the env/patch scripts and the log.
- `scripts/write_results.py`: writes a results file in the spec's contract,
  with provenance.
- `assets/log_template.md`: `REPRODUCTION_LOG.md` skeleton.
- `references/official-repo-workflow.md`: submodule + patches, environments,
  porting old code, security.
- `references/running-experiments.md`: smoke runs, Stage 0, compute estimates,
  launching and monitoring on local GPUs, honest results.

• SKILL.md 10 KB
• assets/log_template.md 1 KB
• references/official-repo-workflow.md 6 KB
• references/running-experiments.md 4 KB
• scripts/scaffold_paper.py 6 KB
• scripts/write_results.py 4 KB