Przejdź do treści
Biblioteka skilli
Dane i analityka · Analiza

srm-check

/srm-check

Sprawdza dane eksperymentu przed analizą wyników, czy podział na grupy nie ma rozjazdu próby SRM.

Licencja
MIT
Ostatnia zmiana repozytorium
26.08.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
2
Skrypty
1
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 26.08.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 1 skrypt w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/srm.py

Co robi i kiedy

Sprawdza dane eksperymentu przed analizą wyników, czy podział na grupy nie ma rozjazdu próby SRM.

Wyzwalacze z opisu

variantgrouptreatmentcontrolarmexperiment_groupbucketA/B test
Oryginalny opis (EN)

The randomization gate before reading any experiment result. Detect Sample Ratio Mismatch (SRM) in experiment data before any analysis proceeds. Auto-fire when data has columns like "variant", "group", "treatment", "control", "arm", "experiment_group", "bucket", or when the user says "A/B test", "experiment", "treatment vs control", "randomization", "analyze this experiment data".

Jak działa

  1. Purpose
  2. When to Use
  3. Instructions
  4. Anti-Patterns
Sekcje z nagłówków SKILL.md, w oryginale · 170 linii, 1 277 słów
Podgląd SKILL.md
---
name: srm-check
description: >-
  The randomization gate before reading any experiment result. Detect Sample Ratio Mismatch (SRM) in
  experiment data before any analysis proceeds. Auto-fire when data has columns like "variant",
  "group", "treatment", "control", "arm", "experiment_group", "bucket", or when the user says "A/B
  test", "experiment", "treatment vs control", "randomization", "analyze this experiment data".
---
**If the skill install path cannot be resolved** (some sandboxed environments): read the script file(s) from this skill, write a copy into a `scripts/` folder inside the working folder, and run from there. The scripts are self-contained.


# Skill: SRM Check (Sample Ratio Mismatch)

## Purpose
Automatically detect Sample Ratio Mismatch in experiment data before any analysis proceeds. SRM is a randomization integrity check — if the treatment/control split deviates significantly from the expected ratio, the experiment is compromised and results cannot be trusted. This skill acts as a safety gate that blocks analysis when randomization is broken.

## When to Use
Apply this skill when:
1. **Loading any experiment or A/B test dataset** — auto-fire on detection of treatment/control columns (e.g., `variant`, `group`, `treatment`, `arm`, `experiment_group`)
2. **Before any treatment effect calculation** — SRM must pass before comparing outcomes
3. **When the Experiment Analyzer agent starts** — first step of any experiment analysis workflow

This skill auto-fires on experiment data detection. Do NOT wait to be asked.

## Instructions

### What Is SRM?

**Sample Ratio Mismatch (SRM)** occurs when the observed ratio of users in treatment vs. control deviates significantly from the expected ratio. For a 50/50 experiment with 10,000 users, you expect ~5,000 in each group. If you see 5,500 vs. 4,500, something is wrong with randomization.

```
SRM CHECK
━━━━━━━━━━
Expected ratio:  50/50 (or whatever was designed)
Observed ratio:  [actual counts]
Test:            Chi-squared goodness-of-fit
Decision:        PASS (proceed) or BLOCK (halt analysis)
```

**Why SRM matters:** If randomization is broken, treatment and control groups are NOT comparable. Any observed difference in outcomes could be caused by the broken randomization, not the treatment. SRM is the single most important validity check in experimentation.

### Detection Logic

#### Step 1: Identify the experiment column

Scan the dataset for columns that indicate experiment assignment. Look for:
- **Column names:** `variant`, `group`, `treatment`, `control`, `arm`, `experiment_group`, `test_group`, `bucket`, `condition`
- **Column values:** binary (0/1, control/treatment, A/B), or small number of distinct values (< 10)
- **User language:** phrases like "A/B test", "experiment", "treatment vs control", "randomization"
- **Metadata:** check for experiment config files in `.knowledge/experiments/`

**If NO experiment indicators are found:** This skill does not apply. **Do not generate any SRM-related output.** Do not explain why the check doesn't apply, do not document the detection logic, do not mention SRM at all. Simply proceed with the user's request as if the SRM Check skill was never loaded. The user is asking for segmentation or comparison analysis, not experiment validation — give them what they asked for without SRM commentary.

#### Step 2: Determine expected ratio

- **If metadata is available** (experiment config, brief, or user-specified): use the designed ratio
- **If no metadata:** default to equal allocation (50/50 for 2 groups, 33/33/33 for 3 groups, etc.)
- **Ask the user if uncertain:** "This looks like an experiment with [N] groups. Was the intended split equal, or was there a specific allocation ratio?"

#### Step 3: Run the SRM test

Use a chi-squared goodness-of-fit test. The deterministic implementation is bundled in this skill at `scripts/srm.py` (needs pandas, numpy, scipy). Add this skill's `scripts/` directory to `sys.path` and import from it:

```python
import sys
sys.path.insert(0, "<path to this skill>/scripts")  # the scripts/ dir next to this SKILL.md
from srm import srm_check

# observed counts per group
result = srm_check(
    observed_counts=[n_control, n_treatment],
    expected_ratios=[0.5, 0.5],  # or designed ratio
)
# Returns: chi2_stat, p_value, verdict (PASS/WARNING/BLOCK), interpretation

chi2 = result["chi2_stat"]
p_value = result["p_value"]
verdict = result["verdict"]
```

For segmented SRM diagnosis (find WHICH segments have broken randomization):

```python
from srm import srm_diagnose  # same bundled scripts/srm.py module

diag = srm_diagnose(assignments_df, segments=["platform", "country"])
# Returns per-segment SRM results to identify the source of the mismatch
```

#### Step 4: Interpret and decide

| p-value | Verdict | Action |
|---------|---------|--------|
| p > 0.05 | **PASS** | Randomization looks clean. Proceed with analysis. |
| 0.01 < p ≤ 0.05 | **WARNING** | Marginal SRM. Proceed with caution. Flag in report. |
| p ≤ 0.01 | **BLOCK** | SRM detected. HALT analysis. Investigate before proceeding. |

#### Step 5: Temporal stability check (if applicable)

**If the experiment ran for multiple days/weeks, check SRM over time:**

1. **Group data by time period** (day, week, or other natural breakpoint based on experiment duration)
2. **Run SRM check for each period** using the same chi-squared test
3. **Flag any period where p ≤ 0.05** (indicates intermittent SRM)

**Why this matters:** SRM can start mid-experiment due to bugs, infrastructure changes, or experiment modifications. A clean aggregate SRM might mask periods where randomization was broken.

**Output format:**
```markdown
### SRM Over Time

| Period | Expected | Observed | Chi-squared | p-value | Verdict |
|--------|----------|----------|-------------|---------|---------|
| Week 1 | 50/50 | 5,012/4,988 | 0.058 | 0.81 | PASS |
| Week 2 | 50/50 | 5,300/4,700 | 36.0 | <0.001 | BLOCK |
```

If any period shows WARNING or BLOCK, investigate before trusting the aggregate result.

### When SRM is Detected (BLOCK)

If the SRM check returns BLOCK:

1. **HALT all downstream analysis** — do NOT compute treatment effects, segment analyses, or recommendations. Results from a broken experiment are unreliable.

2. **Report the finding clearly:**
```markdown
## SRM CHECK: BLOCKED

| Group | Expected | Observed | Deviation |
|-------|----------|----------|-----------|
| Control | [N] ([X]%) | [N] ([X]%) | [+/- Y%] |
| Treatment | [N] ([X]%) | [N] ([X]%) | [+/- Y%] |

**Chi-squared test:** χ² = [value], p = [value]
**Verdict:** Sample Ratio Mismatch detected. Analysis halted.
```

3. **Suggest investigation paths:**
   - Was there a bug in the assignment logic? (Check if certain user types were excluded from one group)
   - Did the feature fail to load for some users? (Treatment users who never saw the treatment get dropped)
   - Was there a data pipeline issue? (Events from one group not logged properly)
   - Was the experiment stopped and restarted? (Can cause ratio imbalances)

4. **Offer options:**
   - "Investigate the cause of the SRM before analyzing results"
   - "Proceed with analysis anyway (NOT RECOMMENDED — results may be invalid)"
   - "Filter to a time window where the ratio was clean (if SRM started at a specific point)"

### Output Format

Always include the SRM check result at the top of any experiment analysis:

```markdown
## Validity Check: Sample Ratio

| Metric | Value |
|--------|-------|
| Expected ratio | [X/Y] |
| Observed | Control: [N] ([X]%), Treatment: [N] ([Y]%) |
| Chi-squared | χ² = [value] |
| p-value | [value] |
| **Verdict** | **PASS / WARNING / BLOCK** |
```

If PASS: continue analysis.
If WARNING: add a note to all findings: "Note: Marginal SRM detected (p = X.XX). Results should be interpreted with caution."
If BLOCK: halt and report as described above.

## Anti-Patterns

1. **Never skip the SRM check** — even if the split "looks close enough," run the statistical test. A 51/49 split on 100K users is a significant SRM.
2. **Never proceed with analysis after a BLOCK** unless the user explicitly overrides — invalid randomization means invalid results
3. **Never assume equal allocation** without checking — some experiments use 80/20 or 90/10 splits by design
4. **Never check SRM only once** — if the data spans multiple time periods (days/weeks), check SRM per-period to detect if the issue started at a specific point
5. **Never dismiss SRM as "close enough"** — SRM is binary. Either randomization worked or it didn't. There is no "close enough" for randomization integrity.
6. **Never explain SRM when it doesn't apply** — if no experiment column is found, produce NO SRM output at all. Exit truly silently without documentation or explanation.
Pliki w paczce (2)
  • SKILL.md 9 KB
  • scripts/srm.py 6 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · ai-analyst-lab/ai-analyst-plugin100MIT
ai-analyst-lab/ai-analyst87MIT

Migawka bazy z 13.11.2026 · wersja paczki 17859453d4b8 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100experiment
    Prowadzi pełny cykl eksperymentu: projekt, analiza mocy, analiza statystyczna, interpretacja, raport i monitoring testów A/B, zaczynając od bramki SRM.
  • 100ab-test-analysis
    Analizuje wyniki testów A/B z kontrolą SRM, CUPED i poprawką na wielokrotne testy oraz liczy wielkość próby.
  • 100experiment
    Odpowiada za analizę i cykl życia eksperymentów: projekt, moc statystyczna, analiza, interpretacja i monitoring testów A/B, z kontrolą SRM na początku.
  • 100ab-test-validity-checklist
    Buduje z propozycji eksperymentu listę kontrolną ważności testu A/B (OEC, moc, jednostka randomizacji, kontrola SRM, podglądanie wyników) wraz z formularzem akceptacji.
  • 90ab-test-analysis
    Prowadzi rygorystyczną statystyczną analizę testów A/B: istotność statystyczna, wykrywanie niezgodności proporcji próby (SRM) i walidacja projektu testu przed startem.
  • 90ab-testing-engineer
    Prowadzi inżynierię eksperymentów A/B i wielowariantowych: hipotezy, metryki, wielkość próby, kontrolę SRM, plan analizy, reguły zatrzymania i rejestr eksperymentów.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.