design-experiment

---
name: design-experiment
description: Plan LLM fine-tuning and evaluation experiments. Use when the user wants to design a new experiment, plan training runs, or create an experiment_summary.yaml file.
---

# Design Experiment

You help users plan experiments for fine-tuning and evaluating LLMs. Create a plan that specifies the complete workflow from training through evaluation, verifies resources, and documents all steps in a structured YAML configuration.

## Your Task

Guide the user through designing their experiment by asking questions, verifying resources, and creating a comprehensive `experiment_summary.yaml` file that documents the complete plan.

## Prerequisites Check

Quick existence check — `claude.local.md` must be present before designing an experiment, because every output path and SLURM default is read from it:

```bash
ls -la claude.local.md
```

**If missing**, stop and tell the user to run `/ck-setup` first — that skill walks them through `claude.local.md.template` interactively. Do not try to write a `claude.local.md` from this skill.

If `claude.local.md` is present but you suspect drift (placeholders not filled in, fields that look stale), suggest the user run `/ck-setup` in validate mode to get a structured health check rather than trying to validate inline here.

### Dependency version check

Run before proceeding to catch stale envs (user pulled new pins but didn't re-run `pip install -e .`):

```bash
python scripts/check_env.py
```

- **Exit 0**: proceed.
- **Exit 1**: show the printed `STALE ENV` table to the user, ask whether to `pip install -e .` first or continue anyway.

## Workflow

Follow the three-stage process:

### 1. Parameter Selection → `param_selection.md`

Guide the user through 9 interactive steps to gather all experiment parameters:
1. Determine experiment location
2. Understand the experiment (scientific question, variables, and whether tabular-to-text data generation is needed)
3. Confirm tool choices (torchtune for preparation, inspect-ai for evaluation)
4. Design training runs (models, datasets, hyperparameters)
5. Design evaluation runs (tasks, epochs, evaluation matrix)
6. Establish naming (experiment name, run names)
7. Verify resources (models, datasets, eval scripts exist)
8. Get approval (validate first, then present)
9. Create files (proceed to generation stage)

**See `param_selection.md` for:**
- Complete question flow for each step
- Auto-detection logic for experiment location
- Resource verification commands
- Conversation patterns

### 2. Validation → `validation.md`

Before presenting plan to user (step 8), validate completeness:
- ✓ All YAML sections present and properly structured
- ✓ All run names follow convention
- ✓ All parameters documented (variables and controls)
- ✓ Evaluation plan is consistent (0-indexed epochs, base vs fine-tuned)
- ✓ **`controls.system_prompt` is set** — the single source for training and eval (no separate eval copy to match)
- ✓ All resources verified (or noted as prerequisites)

**See `validation.md` for:**
- Complete validation checklist
- Common issues to check
- How to handle missing prerequisites

### 3. Experiment Generation → `experiment_generation.md`

After user approves, create output files:
1. `experiment_summary.yaml` - Structured experiment configuration (use `templates/experiment_summary.yaml`)
2. `logs/design-experiment.log` - Human-readable audit trail (see `logging.md`)

Then ask about next steps (scaffold-experiment?).

**See `experiment_generation.md` for:**
- File creation instructions
- YAML formatting guidance
- Next steps conversation pattern
- Prerequisites handling

---

## Cross-Cutting Concerns

### Logging → `logging.md`

**IMPORTANT:** Throughout param_selection and generation, create detailed log at `{experiment_dir}/logs/design-experiment.log`.

**What to log:**
- ✓ Resource verification (ls, du, df commands and results)
- ✓ Prior run searches (if performed)
- ✓ Decisions (naming, recipe, configuration)
- ✓ File creation

**Format:** Plain text with timestamped action entries

**See `logging.md` for:**
- Complete log format specification
- All action types with examples
- Entry structure and timestamp format
- When to log during workflow

### Templates → `templates/`

Reference materials for output generation:
- `templates/experiment_summary.yaml` - YAML schema and structure for experiment plan

---

## Important Reminders

- **BEFORE generating experiment_summary.yaml:** You MUST read `templates/experiment_summary.yaml` first. Do not freestyle the YAML structure — use the template schema exactly.
- **vis_label is required** in evaluation matrix entries for visualization support (see param_selection.md Step 5)
- **Dataset format terminology:** Describe JSON datasets as "JSON with input/output keys" - never invent format type names
- **Use paths from `claude.local.md`** for models, datasets, scratch directories
- **Always verify resources** exist before finalizing plan (log all verification)
- **System prompt has a single source** - set `controls.system_prompt`; it propagates to both training and eval, so parity is automatic (per-task variation: `evaluation.tasks[].system_prompt`)
- **Prompt can be swept** - the user `prompt` defaults to `controls.prompt` but can vary per-task (`evaluation.tasks[].prompt` — eval-only prompt sweep) or per-run (`runs[].parameters.prompt` — a fine-tune trains on its own prompt). Author the structure, don't make the user hand-edit configs. See param_selection.md → **Prompt Sweeps**
- **Epochs are 0-indexed** - Use [0, 1, 2] in evaluation matrix
- **Base models and eval-only checkpoints** use `epochs: null`, **fine-tuned models** use `epochs: [0, 1]`
- **`controls.dataset_type` is required** (`"chat_completion"` | `"text_completion"`) — read by torchtune at training time and propagated to drive chat-template choice at eval time for every run type, including eval-only
- **Document tool choices** in YAML - torchtune for training, inspect-ai for evaluation
- **Handle missing resources gracefully** - note as prerequisites, don't block the plan
- **If inspect-ai task doesn't exist** - note that `create-inspect-task` skill should be run first
- **Generate YAML, not Markdown** - Use structured YAML format with proper indentation

---

## Module Organization

This skill uses the **param_selection → validation → generation** pattern:

| Module | Purpose |
|--------|---------|
| param_selection.md | 9-step interactive workflow |
| validation.md | Completeness checklist |
| experiment_generation.md | Create YAML and log files |
| logging.md | Plain text audit trail specification |
| templates/experiment_summary.yaml | YAML schema and structure |

**Pattern:** Three action verbs (selection, validation, generation) matching scaffold/run skills, plus cross-cutting logging and templates.

**See `README.md` for:** Complete pattern documentation and rationale.

• README.md 6 KB
• SKILL.md 7 KB
• experiment_generation.md 6 KB
• logging.md 13 KB
• param_selection.md 32 KB
• validation.md 9 KB
• references/model_organisms.md 9 KB
• references/scorers.md 7 KB
• references/tabular_to_text_gen.md 15 KB
• templates/experiment_summary.yaml 18 KB