How long does this A/B test have to run?
A test sized after it starts is a test that ends in an argument. This one answers before: at a 3.2% baseline and a 20% lift you need 13,015 visitors per arm, and the sheet shows what happens to that number when you ask for less.
Marketing Starter A/B Test Sample Size free
After you install, this is the model to open.
How Many Visitors Does This Test Need?
- In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
- Click Start from a template and put that name in the search box.
- Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.
The answer
At a 20% minimum detectable effect, and what the neighbouring rows cost:
- Visitors per arm
- 13,015 26,030 in total
- Weeks to run
- 4.8 at full traffic
- To catch a 10% lift
- 49,777 per arm, 18.4 weeks
- To catch a 30% lift
- 6,037 per arm, 2.2 weeks
Halving the effect you want to detect roughly quadruples the traffic you need, every time, and that single fact decides more test programs than any statistical argument. If the team will tolerate four weeks, four weeks buys the ability to detect about a 22% lift and nothing smaller, so the real question is whether a 22% lift is a plausible thing for this change to produce. Raising power from 80% to 90% takes 13,015 to 17,422: a third more traffic to cut your chance of missing a real effect from one in five to one in ten.
The model
One page, its real traffic, and six candidate effect sizes. Each row runs the same power calculation and converts the answer into weeks at your own traffic.
| Current conversion rate | 3.2% |
| Visitors per week | 5,400 |
| Share of traffic in the test | 100% |
| Power / significance | 80% / 0.05 |
| Visitors per arm and weeks to run | calculated per row |
Once it is in your sheet
- The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
- Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
- Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.
Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.
Next question
- What does a margin of error of four points cost?How Many People Do We Have to Ask?
- A 7% margin in January. How often is the year a loss?Will the Retainer Book Cover the Year?
- Plan for the average month or the bad one?How to Split Spend When Conversion Is a Guess
- When does a new hire start paying for themselves?Is the Next Hire Worth It?
- A product launch risk register pricing five ways a launch goes wrong against mitigation cost, from a calendar invite to production fixes. Free template.What Could Go Wrong With This Launch?
- The pipeline says you beat target. Will you?Will the New Business Pipeline Deliver?
Every model like this one, and the method behind them: Statistics in Google Sheets.