Skip to content
Skills Library
Marketing & content · Planning

experiments

/experiments

Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test

Licence
MIT
Last repository change
04.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 04.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/stats.py

What it does and when to use it

Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test

Original author description

Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test

How it works

  1. 1. Pick the method from the traffic
  2. 2. Fix the metric before the start
  3. 3. Wire the flag (A/B only)
  4. 4. While it runs
  5. 5. Close it
  6. Don't
Original SKILL.md headings · 107 lines, 1,002 words
SKILL.md preview
---
name: experiments
description: Design, run and close experiments so every conversion and growth change produces a result the operator can trust. Picks the method from the traffic the change will see (A/B test, holdout, staggered rollout, or a before/after read that is only directional), sizes and times it with stats.py, fixes the primary metric before the start, wires PostHog feature flags and experiments, checks sample ratio and guardrail metrics, and closes the gtm bet from the real counts. Use when a work session opens a bet on the conversion or activation track, when the user asks for an A/B test, split test, price test, "is this change working", "how long should this test run", or sample size; and when a bet's check date arrives. cro and activation decide what to change; measurement owns the events; gtm owns the bet ledger.
---

# experiments

Every change the operator ships is a bet (`gtm.py bet open`). This skill makes the bet answerable:
the right method for the traffic, one metric chosen before the start, and a close that reads real
counts instead of an impression. A test that can't reach an answer in reasonable time is not run as
an A/B test at all.

`<state>` is the store root printed by `gtm.py where`. Run every script from the **product repo
root**, calling the plugin's scripts by absolute path (`scripts/stats.py` is this skill's folder,
`../gtm/scripts/gtm.py` the state engine). Method details and worked examples:
`references/methods.md`.

## 1. Pick the method from the traffic

Start from the rate the change should move and the visitors it will see per day (from PostHog,
`../measurement`):

```
python3 scripts/stats.py days --baseline 0.03 --mde 0.20 --daily 400
```

`--mde` is the relative lift worth detecting (0.20 = +20%). `days` rounds up to whole weeks
(weekdays and weekends behave differently) and splits alpha across the comparisons when `--arms`
is above 2. It refuses a lift the rate can't reach (baseline × (1 + mde) must stay below 1).
Keep `per_arm` from the output: the close needs it. Read `run_it`:

| situation | method (`--method` on the bet) |
|---|---|
| `run_it: true` (under 8 weeks) and users can be split by a flag | `ab`: A/B test behind a PostHog feature flag |
| too slow, but the unit can be split (pages, prompt topics, accounts, regions) | `holdout`: change half, keep half as the control |
| many similar units, can't split users | `staggered`: roll out in waves, compare waves still waiting |
| none of the above | `before_after`: directional only, said so in every report |

Never run an underpowered A/B test hoping for luck. A bigger change, a higher-traffic page, or a
holdout gives a usable answer sooner. For AI visibility, the holdout is half the prompt topics:
pages for one half, the other half untouched (`--control-metric ai:mention:<bucket>`).

## 2. Fix the metric before the start

Open the bet before any code ships (gtm work_session step 3):

```
gtm.py bet open --id 2026-11-20-pricing-cta --track conversion --channel on_site --tactic page_cro \
  --hypothesis "One plan-picker CTA lifts visit→signup" --metric rate:signup --baseline 0.031 \
  --expected 0.037 --method ab --check 2026-12-18
```

- Conversion bets use `rate:<stage>` metrics (e.g. `rate:signup`, the conversion into that stage,
  written by `../measurement`'s pull). `stage:<id>` metrics are 28-day counts, and `bet open`
  refuses a rate-like baseline on them.
- One primary metric, written in the bet. It is the only one that decides won or lost.
- Name 1-2 guardrail metrics in the hypothesis (e.g. activation rate, refunds, page errors). A
  win that breaks a guardrail is reported as a loss to investigate, never shipped silently.
- The check date is the expected launch date (when a human merges the flagged PR) plus
  `stats.py days`, not the day the bet opens and not the day the result looks good. If exposures
  haven't reached `per_arm` by then, the bet stays open and due; don't close early on a lucky p.

## 3. Wire the flag (A/B only)

- A human creates the feature flag and the experiment in the PostHog UI, with the primary metric
  set to the bet's event. If the agent is asked to use PostHog's API for this, verify the endpoint
  and the key's scope first; the measurement key is read-only by design.
- The agent writes the code path behind the flag in a PR: the variant reads the flag, the control
  is the current behaviour, both fire the same events. A human merges.
- Copy in the variant passes `../writing`. Changes to price or plans need the human's yes before
  the PR (see `../activation`).

## 4. While it runs

- **No peeking.** Don't stop early on a good or bad day. The one exception is a guardrail breach
  (errors, a refund spike): stop, revert, and close the bet with
  `gtm.py bet close <id> --force --void "guardrail breached: <what>"`. A void bet has no result and
  counts toward neither fit nor learnings. Report the breach in the digest.
- **Sample ratio check** after the first few days: `python3 scripts/stats.py srm --a 4000 --b 4600`
  (add `--split` for an uneven plan). `mismatch: true` (p < 0.001) means a broken assignment, not
  a result. Fix the flag, void the bet (`--force --void "SRM: <cause>"`) and open a new one with a
  new id and check date.
- Nothing else changes on the tested page while the test runs. Note any launches in plan.md.

## 5. Close it

On the check date (`bet_close` in the gtm tick):

1. Read the counts per arm from the PostHog experiment: visitors (exposed) and conversions on the
   primary metric.
2. Explain the numbers: `python3 scripts/stats.py test --a 4000:124 --b 4000:150` (rates, lift,
   p-value, 95% interval).
3. Close: `gtm.py bet close <id> --a 4000:124 --b 4000:150 --min-n <per_arm> --source "PostHog
   experiment <name>"`, with `--min-n` set to the planned `per_arm` from `stats.py size`. gtm.py
   compares the variant (b) with the control (a): p < 0.05 with the predicted sign is won, with the
   wrong sign lost. Fewer than `--min-n` per arm, or too few conversions for the z-test
   (`small_counts`), is inconclusive. A/B bets always close from the two arms' counts.
4. Holdout bets close from the metrics store (`gtm.py bet close <id>` once both the metric and the
   control are measured after the check date). `stats.py holdout` shows the arithmetic.
5. Ship the winner (remove the flag in a PR) or revert the loser. Inconclusive means keep the
   control unless the variant is simpler.

Learnings come only through `gtm.py learnings`: a strong-method result the human accepts.

## Don't

- Don't test several changes on one page at once unless it is a planned multi-variant test sized
  for it (`stats.py days --arms 3` splits alpha across the comparisons).
- Don't report a before/after result as proof. Say "directional".
- Don't change price, plans, billing or anyone's data inside a test without the human's approval.
Files in this package (3)
  • SKILL.md 7 KB
  • references/methods.md 4 KB
  • scripts/stats.py 7 KB

Versions

RepositoryScoreLicence
This page · williampsmith/marketing-skills100MIT
OppositeNor/render-lab100MIT
langwatch/langwatch100Apache-2.0
microsoft/vscode100MIT
growthbook/skills100MIT

Database snapshot: 13.11.2026 · package version 3f4484239500 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100auditing-experiments-flags
    Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.
  • 95auditing-experiments-flags
    Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.
  • 100a-b-testing
    Design statistically sound GTM experiments for copy, channels, landing pages, sequencing, pricing, and funnel conversion. Produces hypothesis, sample-size logic, success metrics, test plan, analysis method, and scale/stop/kill recommendation. Use when planning A/B tests, split…
  • 100ab-testing
    When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test…
  • 100gtm-executor
    Converts product portfolio and business model into traceable go-to-market strategies, marketing plans, launch campaigns, and growth experiments. Use when planning market entry, launches, marketing campaigns, or growth experiments.
  • 100aso-experiment
    Open, check and close ASO experiments - a listing, keyword, screenshot, icon or rating prompt change with a hypothesis, baseline, metric and check-after date, judged later as confirmed, failed or inconclusive. Use when the user says "track this change", "start an experiment",…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.