experiments
/experiments
Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test
- Author
- williampsmith ↗
- Repository
- marketing-skills ↗
- Licence
- MIT
- Last repository change
- 04.11.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 1
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 04.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 1 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/stats.py
What it does and when to use it
Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test
Original author description
Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test
How it works
- 1. Pick the method from the traffic
- 2. Fix the metric before the start
- 3. Wire the flag (A/B only)
- 4. While it runs
- 5. Close it
- Don't
SKILL.md preview
--- name: experiments description: Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test, "is this change working", "how long should this test run", or sample size; and when a bet's check date arrives. cro and activation decide what to change; measurement owns the events; gtm owns the bet ledger. --- # experiments Every change the operator ships is a bet (`gtm.py bet open`). This skill makes the bet answerable: the right method for the traffic, one metric chosen before the start, and a close that reads real counts instead of an impression. A test that can't reach an answer in reasonable time is not run as an A/B test at all. `<state>` is the store root printed by `gtm.py where`. Run every script from the **product repo root**, calling the plugin's scripts by absolute path (`scripts/stats.py` is this skill's folder, `../gtm/scripts/gtm.py` the state engine). Method details and worked examples: `references/methods.md`. ## 1. Pick the method from the traffic Start from the rate the change should move and the visitors it will see per day (from PostHog, `../measurement`): ``` python3 scripts/stats.py days --baseline 0.03 --mde 0.20 --daily 400 ``` `--mde` is the relative lift worth detecting (0.20 = +20%). `days` rounds up to whole weeks (weekdays and weekends behave differently) and splits alpha across the comparisons when `--arms` is above 2. It refuses a lift the rate can't reach (baseline × (1 + mde) must stay below 1). Keep `per_arm` from the output: the close needs it. Read `run_it`: | situation | method (`--method` on the bet) | |---|---| | `run_it: true` (under 8 weeks) and users can be split by a flag | `ab`: A/B test behind a PostHog feature flag | | too slow, but the unit can be split (pages, prompt topics, accounts, regions) | `holdout`: change half, keep half as the control | | many similar units, can't split users | `staggered`: roll out in waves, compare waves still waiting | | none of the above | `before_after`: directional only, said so in every report | Never run an underpowered A/B test hoping for luck. A bigger change, a higher-traffic page, or a holdout gives a usable answer sooner. For AI visibility, the holdout is half the prompt topics: pages for one half, the other half untouched (`--control-metric ai:mention:<bucket>`). ## 2. Fix the metric before the start Open the bet before any code ships (gtm work_session step 3): ``` gtm.py bet open --id 2026-11-20-pricing-cta --track conversion --channel on_site --tactic page_cro \ --hypothesis "One plan-picker CTA lifts visit→signup" --metric rate:signup --baseline 0.031 \ --expected 0.037 --method ab --check 2026-12-18 ``` - Conversion bets use `rate:<stage>` metrics (e.g. `rate:signup`, the conversion into that stage, written by `../measurement`'s pull). `stage:<id>` metrics are 28-day counts, and `bet open` refuses a rate-like baseline on them. - One primary metric, written in the bet. It is the only one that decides won or lost. - Name 1-2 guardrail metrics in the hypothesis (e.g. activation rate, refunds, page errors). A win that breaks a guardrail is reported as a loss to investigate, never shipped silently. - The check date is the expected launch date (when a human merges the flagged PR) plus `stats.py days`, not the day the bet opens and not the day the result looks good. If exposures haven't reached `per_arm` by then, the bet stays open and due; don't close early on a lucky p. ## 3. Wire the flag (A/B only) - A human creates the feature flag and the experiment in the PostHog UI, with the primary metric set to the bet's event. If the agent is asked to use PostHog's API for this, verify the endpoint and the key's scope first; the measurement key is read-only by design. - The agent writes the code path behind the flag in a PR: the variant reads the flag, the control is the current behaviour, both fire the same events. A human merges. - Copy in the variant passes `../writing`. Changes to price or plans need the human's yes before the PR (see `../activation`). ## 4. While it runs - **No peeking.** Don't stop early on a good or bad day. The one exception is a guardrail breach (errors, a refund spike): stop, revert, and close the bet with `gtm.py bet close <id> --force --void "guardrail breached: <what>"`. A void bet has no result and counts toward neither fit nor learnings. Report the breach in the digest. - **Sample ratio check** after the first few days: `python3 scripts/stats.py srm --a 4000 --b 4600` (add `--split` for an uneven plan). `mismatch: true` (p < 0.001) means a broken assignment, not a result. Fix the flag, void the bet (`--force --void "SRM: <cause>"`) and open a new one with a new id and check date. - Nothing else changes on the tested page while the test runs. Note any launches in plan.md. ## 5. Close it On the check date (`bet_close` in the gtm tick): 1. Read the counts per arm from the PostHog experiment: visitors (exposed) and conversions on the primary metric. 2. Explain the numbers: `python3 scripts/stats.py test --a 4000:124 --b 4000:150` (rates, lift, p-value, 95% interval). 3. Close: `gtm.py bet close <id> --a 4000:124 --b 4000:150 --min-n <per_arm> --source "PostHog experiment <name>"`, with `--min-n` set to the planned `per_arm` from `stats.py size`. gtm.py compares the variant (b) with the control (a): p < 0.05 with the predicted sign is won, with the wrong sign lost. Fewer than `--min-n` per arm, or too few conversions for the z-test (`small_counts`), is inconclusive. A/B bets always close from the two arms' counts. 4. Holdout bets close from the metrics store (`gtm.py bet close <id>` once both the metric and the control are measured after the check date). `stats.py holdout` shows the arithmetic. 5. Ship the winner (remove the flag in a PR) or revert the loser. Inconclusive means keep the control unless the variant is simpler. Learnings come only through `gtm.py learnings`: a strong-method result the human accepts. ## Don't - Don't test several changes on one page at once unless it is a planned multi-variant test sized for it (`stats.py days --arms 3` splits alpha across the comparisons). - Don't report a before/after result as proof. Say "directional". - Don't change price, plans, billing or anyone's data inside a test without the human's approval.
Files in this package (3)
- SKILL.md 7 KB
- references/methods.md 4 KB
- scripts/stats.py 7 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · williampsmith/marketing-skills | 100 | MIT |
| OppositeNor/render-lab | 100 | MIT |
| langwatch/langwatch | 100 | Apache-2.0 |
| microsoft/vscode | 100 | MIT |
| growthbook/skills | 100 | MIT |
Database snapshot: 13.11.2026 · package version 3f4484239500 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100auditing-experiments-flags
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup. - 95auditing-experiments-flags
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup. - 100a-b-testing
Design statistically sound GTM experiments for copy, channels, landing pages, sequencing, pricing, and funnel conversion. Produces hypothesis, sample-size logic, success metrics, test plan, analysis method, and scale/stop/kill recommendation. Use when planning A/B tests, split… - 100ab-testing
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test… - 100gtm-executor
Converts product portfolio and business model into traceable go-to-market strategies, marketing plans, launch campaigns, and growth experiments. Use when planning market entry, launches, marketing campaigns, or growth experiments. - 100aso-experiment
Open, check and close ASO experiments - a listing, keyword, screenshot, icon or rating prompt change with a hypothesis, baseline, metric and check-after date, judged later as confirmed, failed or inconclusive. Use when the user says "track this change", "start an experiment",…