stat-ab-testing
/stat-ab-testing
Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.
- Author
- asgard-ai-platform ↗
- Repository
- skills ↗
- Licence
- MIT
- Last repository change
- 06.06.2026
Assessment and security
- Files in this package
- 4
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 06.06.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.
Triggers from the description
Original author description
Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.
How it works
- Framework
- Output Format
- Gotchas
- References
SKILL.md preview
---
name: "stat-ab-testing"
description: "Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'."
metadata:
category: "WP-21 設計/資訊/傳播/公衛"
tags: ["statistics", "ab-testing", "experimentation"]
---
# A/B Testing Statistics
## Framework
```
IRON LAW: Calculate Sample Size BEFORE Running the Test
Running a test without knowing the required sample size leads to two
failures: stopping too early (false positives) or running too long (waste).
Required inputs: baseline conversion rate, minimum detectable effect (MDE),
significance level (α), power (1-β). Calculate BEFORE starting.
```
### Sample Size Formula (Proportions)
```
n per group ≈ (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²
```
Quick reference (α=0.05, power=0.8):
| Baseline Rate | MDE (relative) | N per Group |
|--------------|----------------|-------------|
| 5% | 10% (→5.5%) | ~58,000 |
| 5% | 20% (→6.0%) | ~15,000 |
| 10% | 10% (→11%) | ~15,000 |
| 10% | 20% (→12%) | ~4,000 |
### Testing Approaches
| Approach | How It Works | Best When |
|----------|-------------|-----------|
| **Frequentist (fixed-horizon)** | Set sample size, run to completion, then analyze | Standard practice, well-understood |
| **Bayesian** | Update beliefs with data, compute probability of improvement | Want probability statements ("90% chance B is better") |
| **Sequential testing** | Check results at intervals with adjusted thresholds | Need to stop early if clear winner, or limit downside risk |
### Experiment Design Checklist
1. **Hypothesis**: What do you expect to happen and why?
2. **Primary metric**: ONE key metric (conversion, revenue, retention)
3. **Guardrail metrics**: Metrics that must NOT degrade (page load time, error rate)
4. **Randomization unit**: User, session, or device?
5. **Sample size**: Calculated from baseline, MDE, α, power
6. **Duration**: Account for weekly cycles (minimum 1-2 full weeks)
7. **Stopping rules**: Pre-defined — do NOT peek and stop early without correction
### Analysis Steps
1. Check randomization balance (are groups comparable on pre-treatment metrics?)
2. Calculate observed difference and confidence interval
3. Run significance test (z-test for proportions, t-test for continuous)
4. Check guardrail metrics
5. Interpret with practical significance in mind
## Output Format
```markdown
# A/B Test Design: {Experiment Name}
## Hypothesis
- H₀: {no difference}
- H₁: {expected improvement}
- Primary metric: {metric}
- MDE: {X% relative}
## Sample Size
- Baseline rate: {X%}
- Required N per group: {N}
- Estimated duration: {days/weeks}
## Results (post-test)
| Metric | Control | Treatment | Diff | CI (95%) | p-value |
|--------|---------|-----------|------|----------|---------|
| {primary} | X% | X% | +X% | [X, X] | {value} |
## Decision
{Ship / Don't ship / Extend test} — {rationale}
```
## Gotchas
- **Peeking inflates false positives**: Checking results daily and stopping when p < 0.05 can produce a 30%+ false positive rate. Use sequential testing methods if you need to peek.
- **Novelty effect**: New features may show a lift that fades as users get used to them. Run tests long enough (2+ weeks) to stabilize.
- **Simpson's paradox**: An overall positive result can be negative in every subgroup (or vice versa). Segment by key dimensions.
- **Network effects / interference**: If treatment users interact with control users (social features, marketplace), independence is violated. Use cluster randomization.
- **Statistical significance threshold is arbitrary**: α=0.05 is convention, not truth. For high-stakes decisions (pricing, major UX changes), consider α=0.01.
## References
- For Bayesian A/B testing methodology, see `references/bayesian-ab.md`
- For multi-armed bandit approach, see `references/bandits.md`
Files in this package (4)
- SKILL.md 4 KB
- examples/sample_scenario.md 4 KB
- references/bandits.md 11 KB
- references/bayesian-ab.md 8 KB
Versions
The database has no other skills named stat-ab-testing.
Database snapshot: 13.11.2026 · package version 2bfe962baa77 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100ab-test-setup
Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B… - 100ab-testing-strategy
Designs A/B tests, writes structured hypotheses, computes required sample sizes and test duration, performs significance analysis (chi-squared, confidence intervals), produces ship/no-ship/inconclusive recommendations, and advises on frequentist vs Bayesian approach selection.… - 100ab-test-setup
Design and analyze A/B tests: sample size, test duration, and statistical significance for conversion experiments. Use when setting up an A/B test, calculating sample size, designing an experiment, or analyzing results. - 100statistical-analyst
Run hypothesis tests, analyze A/B experiment results, calculate sample sizes, and interpret statistical significance with effect sizes. Use when you need to validate whether observed differences are real, size an experiment correctly before launch, or interpret test results with confidence. - 90product-ab-testing
Use this skill when designing and analyzing A/B tests: hypothesis formation, experiment design, statistical analysis, and decision framework. This skill enforces: hypothesis structure, sample size calculation, statistical significance, AA test validation. Do NOT use for:… - 90ab-testing-engineer
Guides experimentation engineering for A/B and multivariate tests—hypothesis framing, primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power, duration and seasonality, allocation and randomization units, SRM checks, instrumentation, analysis plans…