implement-paper
/implement-paper
Implement and run a paper reproduction from its spec (spec/<paper_id>.md) so that the reproducibility tests in tests/<paper_id>/ can pass. Builds on the paper's official GitHub code whenever possible (git submodule pinned at the spec's commit, with our changes as patch files), sets up a per-paper environment, runs smoke tests and released-checkpoint evaluations, estimates compute and asks before full training runs, and writes results/<paper_id>/*.json. If the spec is missing details it looks them up in the official code or the original paper and proposes spec updates, but only edits the spec a
- Author
- MariusAAROS ↗
- Repository
- paper-reproduction ↗
- Licence
- MIT
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 6
- Scripts
- 2
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 2 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/scaffold_paper.py, scripts/write_results.py
What it does and when to use it
Implement and run a paper reproduction from its spec (spec/<paper_id>.md) so that the reproducibility tests in tests/<paper_id>/ can pass. Builds on the paper's official GitHub code whenever possible (git submodule pinned at the spec's commit, with our changes as patch files), sets up a per-paper environment, runs smoke tests and released-checkpoint evaluations, estimates compute and asks before full training runs, and writes results/<paper_id>/*.json. If the spec is missing details it looks them up in the official code or the original paper and proposes spec updates, but only edits the spec a
Original author description
Implement and run a paper reproduction from its spec (spec/<paper_id>.md) so that the reproducibility tests in tests/<paper_id>/ can pass. Builds on the paper's official GitHub code whenever possible (git submodule pinned at the spec's commit, with our changes as patch files), sets up a per-paper environment, runs smoke tests and released-checkpoint evaluations, estimates compute and asks before full training runs, and writes results/<paper_id>/*.json. If the spec is missing details it looks them up in the official code or the original paper and proposes spec updates, but only edits the spec a
How it works
- 0. Preconditions
- 1. Official code first
- 2. Scaffold the workspace
- 3. When the spec is not enough: propose, then ask
- 4. Implement faithfully
- 5. Run: cheap checks yourself, full runs after asking
- 6. Report
- Files in this skill
SKILL.md preview
---
name: implement-paper
description: Implement and run a paper reproduction from its spec (spec/<paper_id>.md) so that the reproducibility tests in tests/<paper_id>/ can pass. Builds on the paper's official GitHub code whenever possible (git submodule pinned at the spec's commit, with our changes as patch files), sets up a per-paper environment, runs smoke tests and released-checkpoint evaluations, estimates compute and asks before full training runs, and writes results/<paper_id>/*.json. If the spec is missing details it looks them up in the official code or the original paper and proposes spec updates, but only edits the spec after the user approves. Use this whenever the user wants to implement, re-implement, code up, set up, port, run or continue a paper reproduction, get the official code of a paper working, or make a paper's reproducibility tests pass, even if they only name the paper or its paper_id.
---
# Implement a paper from its spec
Input: a `paper_id` (or a paper the user names, mapped to its `spec/<paper_id>.md`).
Output: working code in `papers/<paper_id>/`, results in
`results/<paper_id>/<experiment_id>.json`, and an up-to-date
`papers/<paper_id>/REPRODUCTION_LOG.md`, with `pytest tests/<paper_id>` as the
judge.
The spec is the contract. It was written from the paper, and the tests are
derived from it. Your job is to make the code meet the spec faithfully, not to
make the tests green. A reproduction that honestly fails is useful; one that
quietly changed the method, the protocol or the targets is worse than none.
## 0. Preconditions
- `spec/<paper_id>.md` and `tests/<paper_id>/test_<paper_id>.py` exist. If not,
stop and offer the `paper-to-specs-and-tests` skill first.
- Read the **whole spec** and the test file before writing code. Pay most
attention to: §2 (implementation decision and fallback trigger), §4 (setup),
§5 (evaluation protocol), §6 (targets), §7 (results contract), §8
(assumptions).
- If `papers/<paper_id>/REPRODUCTION_LOG.md` already exists, this is a
continuation: read it, pick up from its Status table and open spec questions,
and don't redo finished steps.
## 1. Official code first
Use the official implementation whenever it is usable. It contains the
details papers leave out, and reproductions built on it are both more faithful
and cheaper. Concretely, by the spec's §2 decision:
- **Base on official repo** → bring it in as a submodule and get it running
(porting it to a current stack if needed). Porting working code beats
rewriting it.
- **Official repo as reference only** → write own code in `src/`, but still add
the submodule (or keep a clone in the scratchpad) and read the official code
for every detail the spec leaves open. Cite `official/<file>:<line>` in
comments where you follow it.
- **Own implementation** → first spend a minute checking whether official code
has appeared since the spec was written (the paper's page, the authors'
GitHub, `huggingface.co/papers/<arxiv_id>`). If it has, that is a spec
update: propose switching (step 3). Otherwise implement in `src/`.
Changing strategy (e.g. the spec's fallback trigger fires and you want to drop
the official code) is a spec change too. Gather the evidence, then ask.
Read `references/official-repo-workflow.md` before touching the official code.
It covers the submodule + patches workflow, the per-paper environment, porting
old code, and reading third-party code safely.
## 2. Scaffold the workspace
```bash
python .claude/skills/implement-paper/scripts/scaffold_paper.py <paper_id> \
[--official-url <github url>] [--sha <commit>] [--python <version>]
```
It creates `papers/<paper_id>/` with `official/` (submodule at the spec's
pinned SHA, read from the spec header when `--sha` is omitted), `patches/`,
`src/`, `configs/`, `tests/`, `scripts/{setup_env,apply_patches,save_patch}.sh`
and `REPRODUCTION_LOG.md`. Then edit `scripts/setup_env.sh` to install what the
code needs and run it. The `.venv` lives in `papers/<paper_id>/.venv`, so each
paper gets the library versions its code needs.
Layout rules that keep reproductions reviewable:
- Changes to upstream code exist **only** as `patches/NNNN-<name>.patch`
(create them with `scripts/save_patch.sh <name>`). One concern per patch.
- Own code, wrappers and entry points go in `src/`, configs in `configs/`
(one per `experiment_id`), and unit tests of own code in `papers/<paper_id>/tests/`.
- Data, checkpoints and run outputs go outside `papers/` (`data/`,
`checkpoints/`, `runs/`, all gitignored), never inside `official/`.
## 3. When the spec is not enough: propose, then ask
You will hit details the spec doesn't settle: a missing hyperparameter, an
ambiguous preprocessing step, an official config that disagrees with the
paper, a target that looks mis-copied. Resolve them in this order:
1. **Look it up.** Check the official code first (configs, defaults, training
script), then the paper itself. Fetch it again if needed, following the
`paper-to-specs-and-tests` skill's `references/fetching.md` (published
version first, the arXiv LaTeX source for tables). Note exactly where the
answer is.
2. **Write the proposal**: the spec section, the current text, the proposed
text, the source, and what it affects (code only, or targets/tests too).
3. **Ask the user before editing the spec.** Show the proposal as a short
diff in your message, then ask with AskUserQuestion. Options: *Apply*,
*Apply as an assumption* (goes in §8 with its uncertainty), *Keep the spec
as is*. Group related questions, up to 4 per call. Don't ask one at a time
if several are pending.
4. **Until the user answers**, don't block on it when you can avoid it. Use
the proposed value provisionally if the work is cheap to redo (it shows up
as `provisional` in the log's Spec questions table), and continue with
other parts. Block only when a wrong guess would waste real compute: never
launch a full run on a provisional value.
5. **After approval**, edit the spec, append a line to its `## Change log`
section (create it at the end if missing: date, section, change, source,
"approved by user"), and mark the question `applied` in the log. If the
change touches §6 (targets, tolerances, claims), update `TARGETS`/`CLAIMS` in
the test file to match and run
`python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <paper_id>`.
Why so strict: the spec and tests are the independent yardstick. If the
implementer can change them freely, the yardstick moves toward whatever the
code produces, and a passing test no longer means anything. The user has to
own every change to what "reproduced" means.
Spec changes covered by this rule include target values, tolerances,
experiments, the protocol, the implementation strategy, the pinned commit and
filled gaps. Don't ask about choices that are purely about code (file layout,
refactors, logging), and don't put them in the spec.
## 4. Implement faithfully
- **Follow the spec, not your taste.** Don't fix what looks like a flaw in the
method, swap in a "better" optimizer, or modernize the architecture. If the
official code and the paper disagree, that is a spec question (step 3).
- **Own code mirrors the paper**: name things as the paper does, reference
equations (`# Eq. (3)`) and sections in comments, and keep the method's
structure visible. Add focused unit tests in `papers/<paper_id>/tests/` for
the parts that are easy to get subtly wrong: loss terms on hand-computed
inputs, masking, shapes, schedules. When official code exists, compare your
module's output against it on the same small input.
- **Results**: every experiment ends by writing its result file with
```bash
python .claude/skills/implement-paper/scripts/write_results.py --paper-id <paper_id> \
--experiment <experiment_id> --per-seed <metric>=v1,v2,v3 \
--implementation official|official-modified|own --config <path> --run-dir <dir> --notes "<deviations>"
```
It records provenance (bench commit, official commit, patch hash). Use the
metric keys and units of spec §7 exactly. A unit mismatch (fraction vs %) is
the most common way to fail a correct reproduction.
- **Log as you go**: every patch and deviation goes in the log's Deviations
table, every spec question in Spec questions, and environment facts in
Environment.
## 5. Run: cheap checks yourself, full runs after asking
Read `references/running-experiments.md`. In short:
1. **Smoke run** of each experiment pipeline on tiny data, written under a
`_smoke` experiment ID. Measure throughput here.
2. **Stage 0**: when checkpoints are released, evaluate them with the official
eval code and run the tests. If they don't match the paper, fix the
eval/data pipeline before any training.
3. **Compute estimate** per experiment from measured throughput. Show the user
a table and **ask which full runs to launch** (AskUserQuestion). Training
jobs hold shared GPUs for hours or days, so that is the user's call.
4. **Full runs** on idle local GPUs (check `nvidia-smi`), detached with
`nohup`, one run dir per run under `runs/<paper_id>/<experiment_id>/`,
each recorded in the log.
5. After each run: evaluate on the spec's protocol, write results, then run
`python -m pytest tests/<paper_id> -v`.
A failing test is information, not something to optimize away. Never widen a
tolerance, select checkpoints or hyperparameters on the test set, drop bad
seeds, or rerun until a seed lands in range. Look for the cause; if none is
found, report the miss with what you checked.
## 6. Report
End each session with a short report to the user, and update the log's Status
table to match:
- per experiment: stage reached, and local value vs paper value ± tolerance
(from the pytest output)
- the implementation route actually used, and the patches and deviations that
can affect numbers
- spec questions: applied, pending (what you need from the user), provisional
- what is still to run, with the compute estimate, and anything blocked
## Files in this skill
- `scripts/scaffold_paper.py`: creates `papers/<paper_id>/`, adds the official
submodule at the pinned SHA, and writes the env/patch scripts and the log.
- `scripts/write_results.py`: writes a results file in the spec's contract,
with provenance.
- `assets/log_template.md`: `REPRODUCTION_LOG.md` skeleton.
- `references/official-repo-workflow.md`: submodule + patches, environments,
porting old code, security.
- `references/running-experiments.md`: smoke runs, Stage 0, compute estimates,
launching and monitoring on local GPUs, honest results.
Files in this package (6)
- SKILL.md 10 KB
- assets/log_template.md 1 KB
- references/official-repo-workflow.md 6 KB
- references/running-experiments.md 4 KB
- scripts/scaffold_paper.py 6 KB
- scripts/write_results.py 4 KB
Versions
The database has no other skills named implement-paper.
Database snapshot: 13.11.2026 · package version 4fe531cf169d (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100paper-to-specs-and-tests
Turn a scientific paper into a reproduction spec (spec/<paper_id>.md) and executable pytest reproducibility tests that check whether local experiments recover the paper's tables and main results. Fetches the published version of the paper (falling back on arXiv), finds and… - 100test-paper
Assess a paper reproduction. Runs the paper's reproducibility tests (tests/<paper_id>/) against the local results (results/<paper_id>/), checks that the results are valid (provenance, stale or fake results, edited targets), gives a verdict (reproduced / partially / not… - 100implement-paper-from-scratch
Guides you through implementing a research paper step-by-step from scratch. Use when asked to implement a paper, code up a paper, reproduce research results, or build a model from a paper. Focuses on building understanding through implementation with checkpoint questions. - 100reproduce
End-to-end paper reproduction from arxiv URL through smoke runs to replication experiments. Handles missing or partial official code, missing training scripts, missing hyperparameters, and private datasets via similar-public-dataset substitution. Use when the user asks to… - 100paper-code-audit
Perform three-way cross-validation of a research paper by comparing the original paper text (via Zotero), official code implementation (via Git), and personal Obsidian notes. Use this skill whenever the user wants to "audit a paper", "verify paper claims against code",… - 96paper-reproduce
Assess reproducibility of an embodied AI paper and create an evidence-linked implementation and experiment plan. Use for reproduction planning, artifact audits, or identifying missing experimental details; it does not imply permission to run costly training or control hardware.