Skip to content
Skills Library
Data & analytics · Automation

ab-tests

/ab-tests

Use when a user wants to run, check or end a regular A/B test of paywalls or flows through the Adapty CLI — plan it, create the draft, change it, start it, read its results, stop it with a winner, replace a running test with a changed one, archive it, or see which tests an app has. Triggers on "run an A/B test of these two paywalls", "test my new flow against the old one", "split traffic 50/50", "start the test", "stop the test, B wins", "end this experiment", "change the weights of my test", "which tests are running", "archive the old tests", "should I stop this test", "how did the test go",

Licence
MIT
Last repository change
12.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
7
Deprecation

No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when a user wants to run, check or end a regular A/B test of paywalls or flows through the Adapty CLI — plan it, create the draft, change it, start it, read its results, stop it with a winner, replace a running test with a changed one, archive it, or see which tests an app has. Triggers on "run an A/B test of these two paywalls", "test my new flow against the old one", "split traffic 50/50", "start the test", "stop the test, B wins", "end this experiment", "change the weights of my test", "which tests are running", "archive the old tests", "should I stop this test", "how did the test go",

Triggers from the description

run an A/B test of these two paywallstest my new flow against the old onesplit traffic 50/50start the teststop the test, B winsend this experimentchange the weights of my testwhich tests are running
Original author description

Use when a user wants to run, check or end a regular A/B test of paywalls or flows through the Adapty CLI — plan it, create the draft, change it, start it, read its results, stop it with a winner, replace a running test with a changed one, archive it, or see which tests an app has. Triggers on "run an A/B test of these two paywalls", "test my new flow against the old one", "split traffic 50/50", "start the test", "stop the test, B wins", "end this experiment", "change the weights of my test", "which tests are running", "archive the old tests", "should I stop this test", "how did the test go",

How it works

  1. How you sound
  2. CLI or MCP
  3. The CLI
  4. The effect card: before start, stop, replace, archive
  5. Results and the leader
  6. A new placement
  7. Workflow
  8. Never
  9. Feedback note
  10. Anything not covered here
Original SKILL.md headings · 408 lines, 4,904 words
SKILL.md preview
---
name: ab-tests
description: Use when a user wants to run, check or end a regular A/B test of paywalls or flows through the Adapty CLI — plan it, create the draft, change it, start it, read its results, stop it with a winner, replace a running test with a changed one, archive it, or see which tests an app has. Triggers on "run an A/B test of these two paywalls", "test my new flow against the old one", "split traffic 50/50", "start the test", "stop the test, B wins", "end this experiment", "change the weights of my test", "which tests are running", "archive the old tests", "should I stop this test", "how did the test go", "who is ahead".
---

# A/B tests: plan, start, stop, replace

A regular A/B test splits the users of one audience at one placement between two or more
paywalls, or between two or more flows. This skill owns the whole life of such a test through
`$ADAPTY ab-tests`: plan, create, update, start, stop, replace, archive. **A start, a stop, a
replace and an archive change what real users see and cannot be undone.** Those four run only after
the user has seen their exact effect and said yes.

It does not own:

- **Building a flow** → `flow-generator`. A flow joins a test only once it is `published`.
- **A paywall's design** → the dashboard's Paywall Builder, or the app's own code for a custom
  paywall. A paywall that differs only in the products it sells you create here (Workflow, step 1).
- **Who sees what on a placement without a test** (segments, audiences) → `audiences`. It also
  makes the segment a test starts on, and the row a test starts on when the row must not go to the
  top (Workflow, step 4).
- **The app code that fetches a new placement** → `adapty-integration`. You create the placement
  ("A new placement"); a test on it reaches no user until the app fetches it.
- **Onboarding tests** — not in the CLI. The user reads them in the dashboard:
  https://adapty.io/docs/add-audience-paywall-ab-test.
- **One test across several placements** (a Crossplacement test) — neither the CLI nor the MCP server can make one:
  `ab-tests create` takes one placement, and a placement write keeps crossplacement entries as
  stored. Say so when asked, and send the user to the dashboard, where a second placement makes
  the test one: https://adapty.io/docs/run_stop_ab_tests.
- **Which test to run, and what to test after a stop** → `growth-advisor`. When it hands a test
  over, its brief settles the plan; read each id again before the draft.
- **Apple Search Ads product-page tests** → `ads-manager`.

## How you sound

- **"I" for what you do, "you" for what they do.** Lead with the answer, end on one next step; no
  opener, no recap. Name a skill only when the user must act on it themselves.
- **Name users and content, not ids:** "German users", "the Annual paywall", "test Price test".
  Ids only inside a command.
- **A fixed block where a decision is made (the effect card), one line otherwise.** Say each thing
  once.
- **Bold the one thing that cannot be undone,** at most once per block.
- **Be honest about scope:** say what you did not read (shared tests, a period of the results) and how the
  user can check.
- Full, clickable links. A docs page is read as its `.md` URL; the user gets the same URL without `.md`. Reply in the user's language; the dashboard's own words stay ("Modify" is
  `replace`).

## CLI or MCP

This skill reaches Adapty through the Adapty CLI or the Adapty MCP server. Its commands are written
for the CLI, and over MCP each one is a tool call. Before the first call, check both, pick one way,
and keep it for the run:

- **The CLI is ready** when `adapty auth whoami` shows an account. Install nothing for this check.
- **MCP is ready** when you hold Adapty's tools (names ending in `apps_list` and `auth_whoami`,
  after any prefix your client adds) and `auth_whoami` shows an account.

| CLI ready | MCP ready | Use |
|---|---|---|
| yes | no | the CLI |
| no | yes | MCP |
| yes | yes | the way this skill prefers (the line after this section's example) |
| no | no | ask which one to set up (below) |

**Over MCP, run each command as its tool.** `$ADAPTY <topic> <command>` is the tool whose name ends
in `<topic>_<command>`, dashes and spaces turned into underscores: `ab-tests list` is
`ab_tests_list`, `flows config get` is `flows_config_get`. The flags become the tool's parameters;
take their names and shapes from its schema (`--app` is `app_id`, a flag that may repeat is one
list). The tools return JSON, so drop `--json`. `--yes` has no parameter: the yes this skill asks the
user for still comes before the call. Skip the `$ADAPTY` resolution, `--help` probes, `auth login`
and `auth status`: a command exists when its tool is in your list, and the client signs in.

**`feedback` is the tool `feedback_send`,** with the same fields; its `{"status": "sent"}` answer
counts as `Feedback sent.`

**A command with no tool, and a script that runs `$ADAPTY`, need the CLI** (`segments create` has
no tool). Run that step with the CLI when it is ready. Otherwise say it needs the CLI and offer to
set it up.

**When a call fails one way,** a read may run once the other way if that way is ready. After a
write fails or times out, read the state again the way that wrote; never repeat the write the
other way.

**Neither ready:** ask which one to set up, the one this skill prefers first, then give only that
one's steps. Use your question tool if you have one; otherwise ask in one message with numbered
options.

- **The CLI:** run this skill's CLI resolution, then `adapty auth login`, which opens the browser for
  the user's sign-in. The Claude app also needs Adapty's domains allowed first:
  https://adapty.io/docs/developer-cli-cowork
- **MCP:** the server is `https://mcp.adapty.io/mcp`, and the user signs in in the browser. The
  `adapty-skills` plugin adds it, so when your client says the server needs sign-in, only the
  sign-in is left.
  - Claude Code: `/mcp`, pick `plugin:adapty-skills:adapty`, **Authenticate**. Without the plugin,
    first `claude mcp add --transport http --scope user adapty https://mcp.adapty.io/mcp`.
  - Codex CLI: the plugin's connect prompt signs in. Without the plugin:
    `codex mcp add adapty --url https://mcp.adapty.io/mcp`, then `codex mcp login adapty`.
  - The Claude app: connect it on the plugin's **Connectors** tab. Without the plugin: Settings →
    Connectors → **Add custom connector**, with the URL above.
  - Any other client: its own way to add a remote MCP server that signs in with OAuth.

  If the tools do not appear in this session, the user starts a new one: say so and stop.

Example of the question's shape, not words to copy:

> I can reach your Adapty account in two ways. Which one should I set up?
> 1. The Adapty CLI (better for this): I install it, and you sign in once in the browser.
> 2. The Adapty MCP server: you add it to your AI tool and sign in; nothing to install.

**This skill prefers MCP** when both are ready.

## The CLI

Resolve `$ADAPTY` once and use it for every call:

```bash
adapty ab-tests --help >/dev/null 2>&1 && ADAPTY="adapty" || {
  npm i -g adapty@latest >/dev/null 2>&1 && ADAPTY="adapty" || ADAPTY="npx --yes adapty@latest"; }
```

Older CLI releases have no `ab-tests` commands. If `$ADAPTY ab-tests --help` still
reports no such topic, try `npx --yes adapty@beta ab-tests --help` once; if that fails too, tell
the user their CLI has no `ab-tests` commands yet and the test is made in the dashboard (link
above). Do not compare version numbers. `--yes` on `npx` is required: without it `npx` waits for a
confirmation nobody gives. In `zsh`, run `setopt shwordsplit` before using a multi-word `$ADAPTY`.
Write a literal `adapty` in a command you show the user.

**The results commands** (`ab-tests metrics`) need a newer CLI than the life commands. Before you
read a result, run `$ADAPTY ab-tests metrics --help`. If it reports no such topic: when `$ADAPTY` is
an installed `adapty`, run `npm i -g adapty@latest` once and probe again; then try
`npx --yes adapty@beta ab-tests metrics --help` once (use that form for the reads if it works). If
every probe fails, say the user's CLI has no A/B results commands yet and the results are in the
dashboard (https://adapty.io/docs/results-and-metrics.md), and read no numbers. Do not compare
version numbers. The life commands work without them.

Every command takes `--app <id>`, the **Adapty** app UUID from `$ADAPTY apps list`. Read with
`--json`.

| Command | Flags |
|---|---|
| `ab-tests list` | `--type paywall\|flow` (default **flow**: a paywall test is missing from a default list, so ask for both), `--state live\|draft\|completed\|inactive\|archived`, `--placement` (may repeat), `--segment` (may repeat: the tests of that segment's audience, drafts with that planned segment too), `--search`, `--page`, `--page-size` (use 100) |
| `ab-tests get <id>` | the test and its arms |
| `ab-tests metrics total <id>` | `--type paywall\|flow` (required: the `type` of `ab-tests get`): each arm over the test's whole life: weight, revenue, P2BB, interval, p-value. **Read the leader here** |
| `ab-tests metrics list` | `--type`, `--ids <id>,<id>` (required): the lifetime numbers of several tests of one type, a Total line for each test and a line for each arm; `--csv` |
| `ab-tests metrics get <id>` | `--type`, then `--date-from`, `--date-to`, `--date-type purchase_date\|profile_install_date`, `--granularity`, `--group-by country\|store`, `--filter key=value`, `--timezone`, `--csv`: one window, by period. A paywall test needs the dates and the date type; a flow test with none gives lifetime numbers |
| `ab-tests metrics values <id>` | `--type`: the keys and values that `--filter` takes |
| `ab-tests create` | `--title`, `--goal`, `--placement` (required, of the test's kind: paywall or flow), `--segment <id>` or `--all-users` (a plan only), exactly one of `--paywalls '<json>'` / `--flows '<json>'` |
| `ab-tests update <id>` | the same flags as create (`--placement` required). **The whole test is sent again:** a flag you leave out is cleared (goal, planned audience). A started test changes only title and goal: send the same arms, placement and audience again (`--all-users` when `segment_ids` is `[]`, else `--segment <id>`); a completed or archived test cannot change |
| `ab-tests start <id>` | only a draft; `--placement`, exactly one of `--segment <id>` / `--all-users`, `--yes` |
| `ab-tests stop <id>` | exactly one of `--winner <id>` (the `paywall_id` or `flow_id` of an arm of the test), `--other <id>` (the `paywall_id` or `flow_id` of other content, not an arm), `--no-content` (not on All users), `--yes` |
| `ab-tests replace <id>` | `--title`, `--goal`, `--placement` (required), exactly one of `--paywalls` / `--flows` with the same number of arms as the live test, `--yes`; no audience flag |
| `ab-tests archive <id>` | `--yes` (an archived test cannot be archived again) |
| `flows get <id>` | `status`: `draft`, `dirty`, `publishing`, `published` or `publication_failed` (`flows list` may leave it out) |
| `paywalls create` | `--title`, `--product-id <id>` once per product; a paywall with no design of its own |
| `placements create` | `--title`, `--developer-id`, `--audiences '<json>'` ("A new placement") |

Arms are a JSON array: `[{"paywall_id":"<id>","weight":50,"title":"A"}, …]`, or the same with
`flow_id`; `title` is optional. The weights sum to 100; the server checks that and the number of
arms. A `get --json` test has `ab_test_id`, `title`, `goal`, `type` (`paywall` or `flow`), `state`,
`placement` (`placement_id`, `title`, or `null`), `segment_ids` (`[]` is All users, `null` is none
planned), `priority`, `started_at`, `stopped_at`, and `paywalls[]` or `flows[]` with `weight` and
`title`. `state` is one of the five above.

Without `--yes`, the four one-way commands ask at a terminal; in a pipe or with `--json` they
refuse with exit 6 `confirm_required` and send nothing. Exit 4 is the server's refusal, with its
own text. These come back often:

| The server says | It means | You do |
|---|---|---|
| `The sum of the weights must equal 100, but has N.` | the arms | fix the weights |
| `Flow(<id>) must be published before adding to AB test.` | an unpublished flow | `flow-generator` publishes it first |
| `Paywalls cannot be changed in non-draft ab tests.`, `Placement cannot be changed in non-draft ab tests.` | an update on a started test changed arms or placement | send the arms, placement and audience as read, or use `replace` |
| `AB Test from default audience can not be stopped.` | `--no-content` on All users | `--winner` or `--other` instead |
| `Only a draft can start. To run it again, make a copy: get, then create.` | a start of a test that is not a draft | offer a copy: `get` the old test, `create` a new one; for a live test also `replace` |
| `Completed and archived ab tests cannot be changed.`, `Audience cannot be changed in non-draft ab tests.` | an update of a finished test, or of the audience of a started one | a new test; for a live test `replace` or the title and goal only |
| `ABTest should contain similar types for paywalls and placement.` | a paywall test on a flow placement, or the reverse | pick a placement of the test's kind |
| `The number of variants cannot be changed in a live ab test.` | a replace with another number of arms | send the same number |
| `This test is already archived.` | an archive of an archived test | nothing to do |
| `Archived paywall can not be used.`, `Flow is deleted.` | a create or replace with archived or deleted content | pick other content |
| `AB Test to stop must be live.` | a stop or replace of a test that is not live | say so; a new test is the way |
| `Live Ab Test can not be deleted.` | an archive of a live test | stop it first (a stop has its own card) |

## The effect card: before start, stop, replace, archive

1. **Read the test:** `ab-tests get <id> --json`. For a start, stop or replace also read the
   placement: `placements get --app <app> <placement> --json` (`audiences[]` with `content_type`,
   `segment_ids`, `priority`, the `paywall_id` or `flow_id`, or for a test `ab_test_title`). For an
   archive, read the placement too when the test has one, to say where it sat. Turn ids into names
   with `paywalls list`, `flows list`, `segments list --page-size 100`. **Keep this placement read**:
   it is what the user saw.
2. **Show the card** and wait:

   > Start "Price test" on the Main placement for German users.
   > - Now: German users see the Monthly paywall.
   > - From now: they see the Monthly paywall (50%) or the Annual paywall (50%), a new draw for each
   >   user. A change for a segment can take up to 20 minutes to reach every user.
   > - Stops: the Monthly paywall stops for German users; the test "Old test" running there
   >   becomes completed.
   > - Moves: German users who also match "Premium users" go into the test.
   > - **Cannot be undone.** Stopping it later does not bring back what it replaced.
   >
   > Start it?

   This shows the shape, not words to copy. Each line says who sees what from when, names what
   stops, and ends on what cannot be undone. Per action:
   - **Start.** The audience is the placement row with the same segment (`[]` is the default row).
     A segment with no row gets a new row at the top, so users of this segment who match another
     audience move into the test: name those audiences. A row that holds a test ends that test: it
     becomes completed. A row that holds a shared test (`cross_ab_test`,
     `flow_cross_ab_test`) is not yours: send the user to the dashboard.
   - **Stop.** Say which content the audience shows afterwards (the winner, the other content, or
     nothing: its row leaves the placement and its users get the next audience they match). The
     test becomes completed and **does not run again**.
   - **Replace.** The live test stops and becomes completed, its results stay on its id. A new test
     with a new id starts on the same audience with new arms, and every user of the audience gets a
     new draw. Another `--placement` moves the audience there (refused on All users).
   - **Archive.** The test leaves the lists and **cannot be unarchived.** Only a test that does not
     run; a live one is stopped first. An archived test cannot be archived again.
3. **Wait for the user's yes.** Only an answer to this card counts, not an earlier "go ahead"
   about the plan. "No", silence or a change of plan: send nothing.
4. **Read the placement again** just before a start, stop or replace and compare its audiences
   (content, segments, order, tests) with the read the user saw. If it differs, show a new card
   for the new situation and ask again. The same goes for the test: if its state changed, start over
   from step 1.
5. **Run the command with `--yes`**, then read the test back with `get --json` and say what now
   runs. After a start or a replace, send the [feedback note](#feedback-note) when it is that
   test's first in this session.

**An unattended run never does these four** (a scheduled run, a script, no user to answer), nor
`paywalls create` or `placements create`, which wait for a yes too. It may list, get, create and
update tests. For the four it stops and reports the card and the command for the
user to run, written with a literal `adapty` and **without** `--yes`: at their terminal the CLI asks
again.

## Results and the leader

For "should I stop this test", "how did it go", "who is ahead": `ab-tests metrics total <id> --type
<type> --json` (the type from `ab-tests get`). Show each arm by name with weight, revenue, P2BB and
interval (`probability`, `low_boundary`, `average_per_1000`, `upper_boundary`), and `p_value` for a
flow test when it is a number. What each number means and when it is not computed:
[references/results.md](references/results.md). Say it where the user needs it, once.

The leader is the first of these that holds, from the lifetime P2BB of the arms in `total` or
`list`:

1. P2BB 0 on every arm: "P2BB is not computed". No arm.
2. No `purchases` on any arm and the same P2BB on every arm: "not computed: no purchase". No arm.
3. No `views` on any arm, whatever the P2BB: "not computed: no views". No arm.
4. Paywall test: the arm with the highest P2BB above 0, said as the arm the dashboard highlights.
   A tie above 0: the arms are level and the dashboard preselects the first arm of the total.
5. Flow test: the arm with a strict top P2BB above 0, said the same way. A tie: the arms are level,
   no arm.

The leader is not a winner. Take it only from `total` or `list`, never from the P2BB of a period
(`metrics get`): a period's P2BB is a running total for a paywall test, so say no trend and never
"led on a day". Ask the user for the winner; stop only on their answer.

## A new placement

When no placement fits, you create it, and the developer ID needs its own yes: it is unique across
every placement in the app whatever its type, cannot be renamed, and the CLI cannot delete it. Take
the ID from the user or the app's code; never invent one silently.

> Ready to create the Settings placement for the test.
> - Developer ID: `settings`, the string the app fetches it with.
> - Everyone sees the Monthly paywall there until the test starts.
> - No user sees it until the app fetches `settings`: that is a code change and an app release.
> - **Permanent.** A wrong ID is spent.
>
> Create it?

This shows the shape, not words to copy. On the yes, create it with arm A on the default
audience, `paywall_id` for a paywall test or `flow_id` for a published flow:

```bash
$ADAPTY placements create --app <app> --title "<title>" --developer-id "<id>" \
  --audiences '[{"content_type":"paywall","paywall_id":"<arm A id>","segment_ids":[],"priority":0}]' --json
```

Then say the app needs the fetch call, and offer `adapty-integration` to write it. A test on this
placement has no users until that release is out.

## Workflow

1. **Plan.** Settle in one message what is missing: paywall test or flow test; the arms (content
   and weight, 50/50 for two); the placement; the audience (All users or one segment); a goal in a
   line. Take content ids from `paywalls list --page-size 100` or `flows list`, and the placement
   from `placements list --page-size 100 --json` (it gives only the id, title and developer ID, and
   may show whether it is live). A paywall test belongs on a paywall placement and a flow test on
   a flow placement: read `placements get` for the `content_type` of its audiences; on a mismatch say so
   and ask. Then check that each piece exists, and make what is missing:
   - **A flow arm:** `flows get <id> --json` shows `status` `published` (`flows list` may leave
     the status out). A `draft` or `dirty` flow (unpublished changes) cannot join a test:
     `flow-generator` publishes it first. `publishing`: wait and read again. `publication_failed`:
     not usable; the flow goes back to `flow-generator`.
   - **A paywall arm that does not exist yet,** when it differs only in its products: take the
     product ids from `products list --page-size 100 --json`, only products the user named, then on
     their yes run `paywalls create --app <app> --title "<title>" --product-id <id> [--product-id
     <id> …] --json`. No user sees it until the test starts. A new product is made in the store
     first, then added to Adapty: say so and wait. A paywall with a new design is the user's to
     make in the dashboard or the app's code: say which and wait. A custom paywall arm (no Paywall
     Builder design) also needs the app to log each view, or the test has no view counts: route that
     to `adapty-integration`, which adds the view logging.
   - **A placement that does not exist:** "A new placement".
   - **A segment:** `segments list --page-size 100`. None matches → invoke `audiences` for the
     segment only, saying it is for an A/B test: it creates the segment, returns its id and writes
     no placement.
2. **Create the draft:** `ab-tests create … --json`. A draft shows to no user, so it needs no
   card. Read it back, send the [feedback note](#feedback-note), and say the arms and weights as
   the user will see them.
3. **Change a draft:** `ab-tests update`, with every flag resent as read.
4. **Start**, only when `state` is `draft` in the read you just made (otherwise see Never), with
   the card. `--placement` and the audience flag name where it runs; planned values on the draft
   change nothing until then. A segment with no row on the placement gets a new row at the top, and
   a start cannot put it anywhere else. When the test must sit below another audience (its users
   keep what they see), first have `audiences` add the segment as a regular row at that position,
   showing the content those users see now, so nobody's screen changes; then start on that row.
5. **Change a started test.** Only the title and the goal: `update`, which keeps the test, its start
   time and its draw; resend the arms, the placement and the audience exactly as `get`
   returned them (arm titles too; `--all-users` when `segment_ids` is `[]`, else `--segment <id>`):
   the audience of a started test never changes. New arms or weights on a live test: `replace`, with its card, with the same number of arms. A completed or archived
   test cannot change: create a new test.
6. **Stop**, with the card, when the user names the winner or the other content, or says "no
   content". Ask which if they only say "stop it". "No content" is impossible on All users: offer
   a winner or other content. **Before the card, read the results** ("Results and the leader"), show
   them, and ask for the winner. Never pick it, never stop on the leader.
7. **Archive** old tests with the card, one at a time.
8. **Close** with what runs now and one next step.

## Never

- **Never start a test that is not a draft.** The server refuses it. Say so and offer a new test
  (`get` the old one and `create` a copy), or `replace` for a live one. A live test is already
  running: there is no start.
- **Never run start, stop, replace or archive without the user's yes to the card of this
  situation**, and never on a placement that changed since they saw it.
- **Never put `--yes` on a command the user will run themselves.** It deletes the question the CLI
  would ask.
- **Never update a placement.** No `placements update`, and no test entry in `--audiences`, to
  start, stop, move or "attach" a test: that write starts and stops tests without a card. Only
  `ab-tests` commands start, stop and replace tests. The one placement write is `placements create`
  for a placement that does not exist, after its card ("A new placement").
  A placement rename or any other placement update goes to `audiences`, which sends every test
  row back as read.
- **Never run a one-way command without `--yes` to "see the effect".** Build the card from your
  two reads.
- **Never guess an id.** Take each one from a list or a read. Never send an arm id you did not read.
- **Never retry a refused write with changed values to get it through.** Say the server's reason in
  plain words and ask.

## Feedback note

**Send Adapty one feedback note per test, right after the first `create`, `start` or `replace` of
it in this session.** It tells Adapty what users test and where the idea came from. A run that only
lists, reads, updates, stops or archives sends none. A run that ends before any test exists because
this skill or the CLI stopped you sends one then. Write it in English, in this shape: the shape to
follow, not words to copy:

> Testing: a free trial on the yearly plan against no trial. Source: growth-advisor. Did: created a
> draft.

> Testing: not clear. Source: not given. Did: started a draft.

> Testing: three plans against two. Source: own data, conversion by plan. Did: replaced a live
> test. Got in the way: ab-tests replace failed with http_500 (request id req_7f3a9c), and this
> skill does not say whether a retry is safe.

- **Testing is what the arms differ in, in a few generic words**: the kind of change and its
  direction ("a lower price on the monthly plan", "two paywall designs with the same products", "a
  shorter onboarding"). Take it from what the user said, a brief you were handed, and the arms you
  read; ask nothing and read nothing more for the note. When those do not tell you (arms named A
  and B, nothing said), write "not clear".
- **Source is where the idea came from, as the session shows it**: `growth-advisor` (its
  hypothesis, handed over or quoted), `own data` with the kind of data in a few words (the user
  read their numbers first, here or in the dashboard, or said so), `teardown`, `past test` (a
  follow-up on an earlier test), `other` with a few generic words ("a competitor's paywall"), or
  `not given`. Never ask for it.
- **Did** is one of: created a draft, started a draft, replaced a live test, stopped before a draft.
- **No value and no name from the account or the user's business**: no price, rate, revenue, user
  count, weight or date; no app, paywall, flow, product, placement, segment, brand or competitor
  name; none of the user's words. "A lower price for one country", never which country or how
  much. The ids in `--context` are all Adapty needs.
- **Sentiment is your own outcome, never the user's mood**: `positive`, done the way this skill
  describes, or declined because this skill says it cannot be done; `neutral`, done after a wrong
  turn or a workaround; `negative`, this skill or the CLI is what stopped you. Neutral and negative
  notes add "Got in the way:": the command and its error, or the step of this skill that was
  missing, never a value.
- **`--context` carries the app id, the test id and the command**: `app <app id>, test <test id>:
  create`. A note with no test names what was attempted: `app <app id>: create a paywall test`.
- **Write the note into the command in single quotes**, never from a file or a script, so anyone
  reviewing the call can read what goes out. Inside single quotes the shell runs nothing, so drop
  backticks and write "does not" for "doesn't":

  ```bash
  $ADAPTY feedback --feature ab-tests --sentiment <positive|neutral|negative> \
    --message '<the note>' \
    --context 'app <app id>, test <test id>: <create|start|replace>'
  ```

- **Tell the user in one line, in the same reply, only when the command printed `Feedback sent.`**:
  in their language, "I also sent Adapty a short note on what this test compares and where the idea
  came from." Any other result (an error, a refused or blocked call, `$ADAPTY feedback --help`
  saying there is no such command) gets no line at all. Never retry, never hold up the reply.
- **The user can turn it off**: if they say not to send feedback, send nothing for the rest of the
  session. If they object to something you did, send a note then: what you did, their objection
  quoted in English after "Objection:", and whether they were right.

## Anything not covered here

`$ADAPTY ab-tests <command> --help` for the flags on the installed version; the `adapty-docs`
skill for any dashboard behaviour. Do not guess a flag, a field name or a URL.
Files in this package (2)
  • SKILL.md 29 KB
  • references/results.md 3 KB

Versions

The database has no other skills named ab-tests.

Database snapshot: 13.11.2026 · package version 4fb0bbc4c5ce (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 95adapty-integration
    Use when a user wants to integrate Adapty SDK into a mobile app, set up in-app purchases with Adapty, add a paywall to their app, or move to Adapty from another purchase system. Triggers on "integrate Adapty", "add Adapty to my app", "set up subscriptions", "add a paywall",…
  • 95migrate-placements
    Use when someone wants to move an app from paywall placements to Flow Builder flows — "migrate my placements to flows", "switch my paywalls to flows", "we have 40 placements on paywalls and want flows", or a bulk request across many apps or paywalls. Creates NEW flow placements…
  • 100ab-test-setup
    When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.
  • 86ab-test-setup
    When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions \"A/B test,\" \"split test,\" \"experiment,\" \"test this change,\" \"variant copy,\" \"multivariat...
  • 100running-e2e-tests
    Execute end-to-end tests covering full user workflows across frontend and backend. Use when performing specialized testing. Trigger with phrases like "run end-to-end tests", "test user flows", or "execute E2E suite".
  • 100ab-test-setup
    Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.