Skip to content
Skills Library
Data & analytics · Analysis

stat-ab-testing

/stat-ab-testing

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

Repository
skills ↗
Licence
MIT
Last repository change
06.06.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
4
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 06.06.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

Triggers from the description

should we A/B test thishow many users do we needis the test result conclusivecan we stop the test early
Original author description

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

How it works

  1. Framework
  2. Output Format
  3. Gotchas
  4. References
Original SKILL.md headings · 93 lines, 543 words
SKILL.md preview
---
name: "stat-ab-testing"
description: "Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'."
metadata:
  category: "WP-21 設計/資訊/傳播/公衛"
  tags: ["statistics", "ab-testing", "experimentation"]
---

# A/B Testing Statistics

## Framework

```
IRON LAW: Calculate Sample Size BEFORE Running the Test

Running a test without knowing the required sample size leads to two
failures: stopping too early (false positives) or running too long (waste).

Required inputs: baseline conversion rate, minimum detectable effect (MDE),
significance level (α), power (1-β). Calculate BEFORE starting.
```

### Sample Size Formula (Proportions)

```
n per group ≈ (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²
```

Quick reference (α=0.05, power=0.8):
| Baseline Rate | MDE (relative) | N per Group |
|--------------|----------------|-------------|
| 5% | 10% (→5.5%) | ~58,000 |
| 5% | 20% (→6.0%) | ~15,000 |
| 10% | 10% (→11%) | ~15,000 |
| 10% | 20% (→12%) | ~4,000 |

### Testing Approaches

| Approach | How It Works | Best When |
|----------|-------------|-----------|
| **Frequentist (fixed-horizon)** | Set sample size, run to completion, then analyze | Standard practice, well-understood |
| **Bayesian** | Update beliefs with data, compute probability of improvement | Want probability statements ("90% chance B is better") |
| **Sequential testing** | Check results at intervals with adjusted thresholds | Need to stop early if clear winner, or limit downside risk |

### Experiment Design Checklist

1. **Hypothesis**: What do you expect to happen and why?
2. **Primary metric**: ONE key metric (conversion, revenue, retention)
3. **Guardrail metrics**: Metrics that must NOT degrade (page load time, error rate)
4. **Randomization unit**: User, session, or device?
5. **Sample size**: Calculated from baseline, MDE, α, power
6. **Duration**: Account for weekly cycles (minimum 1-2 full weeks)
7. **Stopping rules**: Pre-defined — do NOT peek and stop early without correction

### Analysis Steps

1. Check randomization balance (are groups comparable on pre-treatment metrics?)
2. Calculate observed difference and confidence interval
3. Run significance test (z-test for proportions, t-test for continuous)
4. Check guardrail metrics
5. Interpret with practical significance in mind

## Output Format

```markdown
# A/B Test Design: {Experiment Name}

## Hypothesis
- H₀: {no difference}
- H₁: {expected improvement}
- Primary metric: {metric}
- MDE: {X% relative}

## Sample Size
- Baseline rate: {X%}
- Required N per group: {N}
- Estimated duration: {days/weeks}

## Results (post-test)
| Metric | Control | Treatment | Diff | CI (95%) | p-value |
|--------|---------|-----------|------|----------|---------|
| {primary} | X% | X% | +X% | [X, X] | {value} |

## Decision
{Ship / Don't ship / Extend test} — {rationale}
```

## Gotchas

- **Peeking inflates false positives**: Checking results daily and stopping when p < 0.05 can produce a 30%+ false positive rate. Use sequential testing methods if you need to peek.
- **Novelty effect**: New features may show a lift that fades as users get used to them. Run tests long enough (2+ weeks) to stabilize.
- **Simpson's paradox**: An overall positive result can be negative in every subgroup (or vice versa). Segment by key dimensions.
- **Network effects / interference**: If treatment users interact with control users (social features, marketplace), independence is violated. Use cluster randomization.
- **Statistical significance threshold is arbitrary**: α=0.05 is convention, not truth. For high-stakes decisions (pricing, major UX changes), consider α=0.01.

## References

- For Bayesian A/B testing methodology, see `references/bayesian-ab.md`
- For multi-armed bandit approach, see `references/bandits.md`
Files in this package (4)
  • SKILL.md 4 KB
  • examples/sample_scenario.md 4 KB
  • references/bandits.md 11 KB
  • references/bayesian-ab.md 8 KB

Versions

The database has no other skills named stat-ab-testing.

Database snapshot: 13.11.2026 · package version 2bfe962baa77 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ab-test-setup
    Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B…
  • 100ab-testing-strategy
    Designs A/B tests, writes structured hypotheses, computes required sample sizes and test duration, performs significance analysis (chi-squared, confidence intervals), produces ship/no-ship/inconclusive recommendations, and advises on frequentist vs Bayesian approach selection.…
  • 100ab-test-setup
    Design and analyze A/B tests: sample size, test duration, and statistical significance for conversion experiments. Use when setting up an A/B test, calculating sample size, designing an experiment, or analyzing results.
  • 100statistical-analyst
    Run hypothesis tests, analyze A/B experiment results, calculate sample sizes, and interpret statistical significance with effect sizes. Use when you need to validate whether observed differences are real, size an experiment correctly before launch, or interpret test results with confidence.
  • 90product-ab-testing
    Use this skill when designing and analyzing A/B tests: hypothesis formation, experiment design, statistical analysis, and decision framework. This skill enforces: hypothesis structure, sample size calculation, statistical significance, AA test validation. Do NOT use for:…
  • 90ab-testing-engineer
    Guides experimentation engineering for A/B and multivariate tests—hypothesis framing, primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power, duration and seasonality, allocation and randomization units, SRM checks, instrumentation, analysis plans…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.