Przejdź do treści
Biblioteka skilli
Dane i analityka · Analiza

ab-test-analysis

/ab-test-analysis

Analizuje wyniki testów A/B z kontrolą SRM, CUPED i poprawką na wielokrotne testy oraz liczy wielkość próby.

Repozytorium
honestab ↗
Licencja
MIT
Ostatnia zmiana repozytorium
03.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
4
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
1
Przestarzałość

Brak sygnałów. Repozytorium zmienione 03.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Analizuje wyniki testów A/B z kontrolą SRM, CUPED i poprawką na wielokrotne testy oraz liczy wielkość próby.

Oryginalny opis (EN)

Analyse A/B test results or design an experiment with statistical rigour (SRM check, CUPED, delta method for ratio metrics, multiple testing, peeking, sample size) using the honestab library. Use when asked to analyse an experiment, compare a control and a treatment group, decide whether to ship a variant, or compute sample size, MDE or test duration.

Jak działa

  1. 0. Tools
  2. 1. Clarify the experiment before computing anything
  3. 2. Get the data into one row per randomization unit
  4. 3. Check sample ratio mismatch first
  5. 4. Choose the method for each metric
  6. 5. Run the analysis
  7. 6. Interpret honestly
  8. 7. Report
  9. Designing an experiment
Sekcje z nagłówków SKILL.md, w oryginale · 107 linii, 824 słów
Podgląd SKILL.md
---
name: ab-test-analysis
description: Analyse A/B test results or design an experiment with statistical rigour (SRM check, CUPED, delta method for ratio metrics, multiple testing, peeking, sample size) using the honestab library. Use when asked to analyse an experiment, compare a control and a treatment group, decide whether to ship a variant, or compute sample size, MDE or test duration.
---

# A/B test analysis

Most wrong experiment conclusions do not come from the wrong formula but from skipped
checks: lost users, a metric computed per event instead of per user, ten metrics tested
without correction, results checked every day. Follow the steps in order.

## 0. Tools
```bash
python -c "import honestab" 2>/dev/null || pip install honestab
```
If installation from PyPI fails, use `pip install git+https://github.com/sofetizm/honestab`.
The CLI is `honestab`; the Python API is `import honestab as hb`.

## 1. Clarify the experiment before computing anything
Find out (from the user, the data or the docs) and write down:
- the **hypothesis** and the **one primary metric** that decides;
- **guardrail** metrics that must not get worse (errors, latency, cancellations...);
- the **randomization unit** (user, device, session) and the planned **split**;
- whether results were **checked during the test** and the test stopped early.

If something is unknown, state your assumption explicitly in the report. Do not pick the
primary metric after looking at the results.

## 2. Get the data into one row per randomization unit
- If the data is per event (views, sessions, orders), **aggregate to one row per unit**
  first. Never run a test where rows are events: events of one user are correlated and
  the p-values come out far too small.
- A metric that is a ratio of event counts (CTR = clicks / views, average check = revenue /
  orders) becomes two columns per unit — numerator and denominator — and is analysed with
  the delta method (`kind: ratio`).
- Check that no unit appears in both groups; report how many if any (and exclude them).
- Missing values: decide explicitly (0 for "did nothing", or exclude) and say which.

## 3. Check sample ratio mismatch first
```bash
honestab srm <n_control> <n_treatment> [--shares 0.5 0.5]
```
If p < 0.001, **stop**: users were lost or added non-randomly and every metric comparison
is biased. Report the mismatch and possible causes (assignment, logging, filters,
redirects, bots) instead of interpreting metrics.

## 4. Choose the method for each metric
| Metric | `kind` | Method |
|---|---|---|
| per-user value (revenue, orders, time) | `mean` | Welch t-test |
| same, with the value of the metric **before** the test | `mean` + `covariate` | CUPED |
| share of users who did something | `proportion` | z-test |
| ratio of sums (CTR, average check, revenue per session) | `ratio` | delta method |
| median, percentile | `median` | bootstrap |

Details and formulas: `references/methods.md`.

## 5. Run the analysis
Write a config and run it:
```yaml
group_col: group        # column with the variant
control: A
treatment: B
expected_split: 0.5     # planned share of the treatment group
metrics:
  - {name: revenue per user, column: revenue, covariate: pre_revenue, role: primary}
  - {name: conversion, column: converted, kind: proportion}
  - {name: ctr, numerator: clicks, denominator: views}
  - {name: errors per user, column: errors, role: guardrail, higher_is_better: false}
```
```bash
honestab analyze data.csv --config experiment.yaml --json
```
Exactly one metric is `primary`; `secondary` metrics get a Holm correction automatically;
`guardrail` metrics block shipping if they get significantly worse.

## 6. Interpret honestly
- Report the **effect with its confidence interval**, not just the p-value:
  "+3.1% (95% CI +0.4% … +5.8%)".
- Not significant means "no effect detected", not "no effect". Always state the
  **MDE** of the experiment (it is in the report): effects smaller than that could be missed.
- Never say "trending towards significance" or re-test with another method until p < 0.05.
- If the test was **stopped early** because the result looked significant, the p-value is
  not valid. Say so; with data in arrival order use `hb.msprt` (always-valid p-values).
- Segment breakdowns (country, platform) are **exploratory**: many segments → false
  positives. Report them as hypotheses for a new test, never as the conclusion.
- Outliers: only a rule fixed in advance and applied to both groups equally (e.g.
  winsorize at the pooled 99.9th percentile). Never remove "strange" users from one group.

Common traps with numbers: `references/pitfalls.md`.

## 7. Report
Use `references/report-template.md`: decision first, then the primary metric with its CI,
guardrails, secondary metrics, checks performed (SRM, data issues), MDE, assumptions and
limitations.

## Designing an experiment
```bash
honestab design --kind proportion --baseline 0.05 --mde-rel 0.05 --daily 20000
honestab design --kind mean --baseline 280 --std 1100 --mde-rel 0.05 --variance-reduction 0.25 --daily 20000
```
- Use the **standard deviation of the metric per unit** from historical data.
- With a pre-period covariate, pass `--variance-reduction` = ρ² (correlation of the
  metric with its pre-period value, squared) — CUPED needs fewer users.
- For a multiplicative effect on a heavy-tailed metric (revenue) the spread of the
  treatment group grows too: pass `std_treatment=std*(1+lift)` to `hb.sample_size` and
  keep a margin of 10–20%.
- Run at least **one full week** to cover weekly seasonality, even if the sample size is
  reached sooner.
- Fix the primary metric, guardrails, sample size and duration **before** the start.
Pliki w paczce (4)
  • SKILL.md 6 KB
  • references/methods.md 3 KB
  • references/pitfalls.md 2 KB
  • references/report-template.md 1 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · sofetizm/honestab100MIT
The-Utopia-Studio/skills100Apache-2.0
tarunccet/pm-skills100MIT
phuryn/pm-skills95MIT
holistics/skills90Apache-2.0
nimrodfisher/data-analytics-skills90MIT

Migawka bazy z 13.11.2026 · wersja paczki 75cad73dc43e (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100experiment
    Projektuje testy A/B: dokumenty hipotez, wielkość próby, feature flagi, analiza istotności, CUPED, wykrywanie SRM i eksperymenty switchback.
  • 100experiment
    Odpowiada za analizę i cykl życia eksperymentów: projekt, moc statystyczna, analiza, interpretacja i monitoring testów A/B, z kontrolą SRM na początku.
  • 100srm-check
    Sprawdza dane eksperymentu przed analizą wyników, czy podział na grupy nie ma rozjazdu próby SRM.
  • 100experiment
    Prowadzi pełny cykl eksperymentu: projekt, analiza mocy, analiza statystyczna, interpretacja, raport i monitoring testów A/B, zaczynając od bramki SRM.
  • 100manage-experiment
    Prowadzi przez każdą fazę eksperymentu Mixpanel: projekt (hipoteza, metryki, wielkość próby, model statystyczny, CUPED), start, monitorowanie (SRM, tempo próby, guardrails) i interpretację wyników z decyzją ship, iterate, kill lub wait.
  • 87srm-check
    Wykrywa niezgodność proporcji próby w testach A/B i blokuje analizę, gdy podział grup nie jest losowy.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.