How many stores does your pilot really need?
Sizing a store pilot on an 18% attach rate and a 12% lift takes 5,194 transactions per arm. Converting that into stores is where most pilots go wrong, and the template does the conversion and then says why the answer is not the number of stores you need.
Operations Intermediate A/B Test Sample Size free
After you install, this is the model to open.
How Big Does the Pilot Have to Be?
- In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
- Click Start from a template and put that name in the search box.
- Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.
The answer
- The arithmetic says
- 5,194 transactions per arm to see 18% become 20.16%
- Converted to stores
- 2.1 per arm at 620 transactions a week over four weeks
- Run instead
- 8 stores per arm: transactions in one store are not independent
- A store is really
- 1 unit one manager, one rota, one decision about the prompt
The till prompt is meant to lift extended-warranty attach from 18%, and the smallest lift worth rolling out is 12% relative, from 18% to 20.16%. Click Run: 5,194 transactions per arm, 10,388 in total. The stores-per-arm column converts that at 620 transactions a store a week over four weeks and says 2.1 stores per arm. Do not run a two-store pilot.
This is the mistake the arithmetic invites, and it is worth understanding rather than just avoiding. The calculation assumes 5,194 independent observations, and transactions inside one store are not independent: they share a manager, a rota, a layout, a local market and a single decision about whether anyone bothers with the prompt. Two stores gives you two independent units dressed up as five thousand, so a good result might be one enthusiastic manager and a bad one might be a store with a staffing gap, and no p-value computed from transaction counts will tell you which.
The minimum stores line carries the working rule: at least eight stores per arm regardless of what the transaction arithmetic says, matched into pairs on size and format and then randomised within each pair. Sixteen stores at 2,480 transactions each is 39,680 transactions, far more than the calculation asked for, and the surplus is the price of having sixteen independent units instead of two.
Read the stores-per-arm column as a floor rather than an answer: if it asks for more store-weeks than eight stores can produce, the pilot needs to run longer, and if it asks for fewer, you still need the eight. The 12% line is also worth pricing against the 8% one: catching a smaller lift takes 11,518 transactions per arm, which is 4.6 stores of pure arithmetic and the same eight in practice, so the cost of chasing a smaller effect here is length rather than width.
What the calculation cannot know is seasonality. A four-week pilot spanning a promotion measures the promotion in both arms and may measure nothing else. To use your own chain, change the attach rate, the transactions per store and the pilot length on the sheet, then put your own baseline into the panel.
The model
It arrives on a tab called Template: How Big Does the Pilot Have to Be, carrying these columns:
- Current attach rate (extended warranty)
with the model computed beside the data:
| Transactions/store over pilot (count) | 2,480 |
Once it is in your sheet
- The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
- Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
- Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.
Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.
Next question
- Should you drop the slow line?Should We Drop This Line?
- Is a random draw a fair way to split the weekends?Assign the Shifts Without Arguing
- How much of the opening date is just waiting?What Sets the Opening Date?
- Should you make more of your most profitable product?What Should We Make This Week?
- How few people can cover a seven-day week?How Few People Can Staff the Week?
- How many units does one delivery wait really need on the shelf?How many units to stock for 90% odds of no stockout?
Every model like this one, and the method behind them: Statistics in Google Sheets.