Skip to content
Skills Library
Data & analytics · Planning

ab-testing-engineer

/ab-testing-engineer

Guides experimentation engineering for A/B and multivariate tests—hypothesis framing, primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power, duration and seasonality, allocation and randomization units, SRM checks, instrumentation, analysis plans (frequentist and Bayesian workflow level), stopping rules, readouts, rollout/kill decisions, and experiment registry hygiene (sound statistical practice only). Use for "A/B test", "A/B testing engineer", "experiment design", "sample size calculator", "statistical significance", "power analysis", "split test", "experiment

Licence
MIT
Last repository change
20.05.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
7
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 20.05.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Guides experimentation engineering for A/B and multivariate tests—hypothesis framing, primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power, duration and seasonality, allocation and randomization units, SRM checks, instrumentation, analysis plans (frequentist and Bayesian workflow level), stopping rules, readouts, rollout/kill decisions, and experiment registry hygiene (sound statistical practice only). Use for "A/B test", "A/B testing engineer", "experiment design", "sample size calculator", "statistical significance", "power analysis", "split test", "experiment

Triggers from the description

A/B testA/B testing engineerexperiment designsample size calculatorstatistical significancepower analysissplit test
Original author description

Guides experimentation engineering for A/B and multivariate tests—hypothesis framing, primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power, duration and seasonality, allocation and randomization units, SRM checks, instrumentation, analysis plans (frequentist and Bayesian workflow level), stopping rules, readouts, rollout/kill decisions, and experiment registry hygiene (sound statistical practice only). Use for "A/B test", "A/B testing engineer", "experiment design", "sample size calculator", "statistical significance", "power analysis", "split test", "experiment

How it works

  1. When to Use
  2. When NOT to Use
  3. Related skills
  4. Core Workflows
  5. When to load references
Original SKILL.md headings · 106 lines, 676 words
SKILL.md preview
---
name: ab-testing-engineer
description: |
  Guides experimentation engineering for A/B and multivariate tests—hypothesis framing,
  primary/secondary/guardrail metrics, design (A/B, A/B/n, MVT), sample size and power,
  duration and seasonality, allocation and randomization units, SRM checks, instrumentation,
  analysis plans (frequentist and Bayesian workflow level), stopping rules, readouts,
  rollout/kill decisions, and experiment registry hygiene (sound statistical practice only).
  Use for "A/B test", "A/B testing engineer", "experiment design", "sample size calculator",
  "statistical significance", "power analysis", "split test", "experiment readout",
  "guardrail metrics", "SRM", "randomization unit", "experiment duration", "multivariate test",
  "holdout", or "experiment analysis". Not for analytics stack build (analytics-data-engineer),
  marketing copy (cmo-advisor), roadmap-only (cpo-advisor), ML model experiments
  (ml-research-engineer-safeguards), or privacy/legal review (compliance-engineer).
---

# A/B Testing Engineer

## When to Use

- Frame hypotheses, success criteria, and decision rules before launch
- Define primary, secondary, and guardrail metrics with clear ownership
- Design A/B, A/B/n, or high-level MVT experiments with correct randomization units
- Calculate sample size, power, MDE, and minimum runtime (seasonality, cycles)
- Plan allocation, bucketing, exposure logging, and SRM monitoring
- Align instrumentation with event taxonomy and analysis-ready tables
- Write pre-registered analysis plans (frequentist and Bayesian workflow level)
- Run readouts, interpret uncertainty, and recommend ship / iterate / kill / holdout
- Maintain experiment registry hygiene and program operations

## When NOT to Use

- Build or operate warehouse pipelines, dbt models, or BI stack → `analytics-data-engineer`, `data-warehouse-engineer`
- Executive dashboards and KPI storytelling without experiment design → `bi-analyst`
- General ML modeling, causal inference beyond experiments, MLOps → `data-scientist`
- Model training/eval A/B for ML systems (offline/online model comparison) → `ml-research-engineer-safeguards`, `data-scientist`
- Marketing copy, campaigns, or channel strategy → `communication-lead`, `cmo-advisor` (if installed)
- Product roadmap, portfolio prioritization without measurement design → `product-management-monetization`, `cpo-advisor` (if installed)
- Weekly metrics review cadence without new experiment → `metrics-review` (if installed), `bi-analyst`
- Legal privacy, consent, or regulatory sign-off for tracking → `compliance-engineer`, `privacy-research-engineer-safeguards`
- Growth experiment ideation without statistical design → `growth-marketer` (if installed), `marketing-analyst` (if installed)

## Related skills

| Need | Skill |
|---|---|
| General ML, causal inference, production model eval | `data-scientist` |
| Warehouse metrics, dbt, analytics pipelines | `analytics-data-engineer` |
| BI dashboards and metric definitions | `bi-analyst` |
| Business metrics and requirements framing | `business-analyst` |
| Product monetization and pricing experiments context | `product-management-monetization` |
| Model governance and ML experiment boundaries | `ml-research-engineer-safeguards` |
| Privacy engineering for event collection | `privacy-research-engineer-safeguards` |
| Compliance controls for tracking and data use | `compliance-engineer` |
| Growth loops and channel tests (strategy) | `growth-marketer` (if installed) |
| Campaign performance reporting | `marketing-analyst` (if installed) |
| Product strategy without experiment ops | `product-strategist` (if installed) |
| Recurring metrics review rituals | `metrics-review` (if installed) |

## Core Workflows

### 1. Hypothesis and metric contract

1. State problem, user segment, and expected mechanism
2. Lock **one** primary metric for the decision (pre-register)
3. List secondary (learning) and guardrail (safety) metrics
4. Define non-goals and segments that are out of scope for decision
5. Agree rollout criteria: win threshold, guardrail breach, neutral band

**See `references/hypothesis_metrics_and_design.md`.**

### 2. Design and power

1. Choose design type: A/B, A/B/n, factorial/MVT (high level), holdout
2. Set MDE from business impact, not statistical convenience
3. Compute per-variant sample size (α, power, one- vs two-sided)
4. Estimate duration: traffic, allocation %, seasonality, full cycles
5. Document exclusions, mutual exclusion with other tests, and cooldown rules

**See `references/sample_size_power_and_duration.md`.**

### 3. Randomization, instrumentation, and SRM

1. Pick randomization unit (user, account, device, session—justify match to analysis)
2. Define bucketing hash, sticky assignment, and cross-device policy
3. Map exposure events to assignment logs; verify join keys
4. Pre-launch QA: assignment distribution, latency, fallback behavior
5. Monitor SRM and invariant checks during the run

**See `references/randomization_instrumentation_and_srm.md`.**

### 4. Analysis, readout, and decision

1. Follow pre-registered analysis plan; intent-to-treat by default
2. Run primary analysis with CIs; pre-defined segment slices only
3. Apply multiplicity policy if multiple variants or metrics drive decisions
4. Compare Bayesian vs frequentist readout needs with stakeholders
5. Produce readout: effect, uncertainty, guardrails, recommendation, risks

**See `references/analysis_readout_and_decision.md`.**

### 5. Governance and program ops

1. Register experiment in catalog (owner, dates, links, status)
2. Enforce ethical constraints: no p-hacking playbooks, no optional peeking without correction
3. Archive artifacts: config, power calc, analysis notebook, decision log
4. Plan ramp, rollback, and long-term holdout if shipping

**See `references/governance_ethics_and_program_ops.md`.**

## When to load references

| Topic | Reference |
|---|---|
| Role boundaries and deliverables | `references/ab_testing_engineer_scope.md` |
| Hypothesis, metrics, design types | `references/hypothesis_metrics_and_design.md` |
| Sample size, power, duration | `references/sample_size_power_and_duration.md` |
| Randomization, events, SRM | `references/randomization_instrumentation_and_srm.md` |
| Analysis, readout, ship/kill | `references/analysis_readout_and_decision.md` |
| Registry, ethics, program ops | `references/governance_ethics_and_program_ops.md` |
Files in this package (7)
  • SKILL.md 6 KB
  • references/ab_testing_engineer_scope.md 4 KB
  • references/analysis_readout_and_decision.md 5 KB
  • references/governance_ethics_and_program_ops.md 4 KB
  • references/hypothesis_metrics_and_design.md 4 KB
  • references/randomization_instrumentation_and_srm.md 5 KB
  • references/sample_size_power_and_duration.md 4 KB

Versions

The database has no other skills named ab-testing-engineer.

Database snapshot: 13.11.2026 · package version 9c3aa1ce008e (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ab-test-setup
    Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B…
  • 100experiment
    The analysis and lifecycle owner for experiments. Full experiment lifecycle: design, power analysis, statistical analysis, interpretation, reporting, and monitoring of A/B tests. Invoke as /experiment. Trigger on "A/B test", "experiment", "treatment vs control", "sample size",…
  • 100experiment-design
    A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common…
  • 100ab-testing
    Use this skill when planning, designing, or analyzing A/B tests and growth experiments. Trigger phrases: "run an A/B test," "design an experiment," "write a hypothesis," "calculate sample size," "build an experimentation backlog," "ICE scoring," "statistical significance,"…
  • 100experiment
    The analysis and lifecycle owner for experiments. Full experiment lifecycle: design, power analysis, statistical analysis, interpretation, reporting, and monitoring of A/B tests. Invoke as /experiment. Trigger on "A/B test", "experiment", "treatment vs control", "sample size",…
  • 100ab-test-analysis
    Analyse A/B test results or design an experiment with statistical rigour (SRM check, CUPED, delta method for ratio metrics, multiple testing, peeking, sample size) using the honestab library. Use when asked to analyse an experiment, compare a control and a treatment group,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.