The winner got more opens. Did it get more clicks?

Eighteen thousand recipients split two ways. The open-rate gap is real at p 0.003 and the click gap on the same send is not, which is the difference between winning a subject line and winning a campaign.

Marketing Starter Statistics free

Install free All 38 statistics tools

After you install, this is the model to open.

Which Subject Line Won?

In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
Click Start from a template and put that name in the search box.
Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.

The answer

Opens: +8.95% 20.0% to 21.8%, p = 0.0028
Clicks: p = 0.47 no verdict on the same send
Click-to-open: 17.9% against 18.5%: the extra opens click less
Full-list gain: 329 opens and no proven extra clicks

One newsletter, two subject lines, 18,350 recipients split almost evenly. Click Run on the opens: 20.022% against 21.814%, a relative lift of 8.95%, with z of 2.99 and a two-tail p-value of 0.0028. Version B genuinely wins on opens, and on the full list that is about 329 extra people opening the email. Notice how small the gap is and how certain the verdict: 1.8 percentage points, decided beyond argument, because nine thousand recipients per arm is a lot of evidence.

That cuts both ways, and the second run is where it bites. Change the two conversion counts to the clicks, 341 against 358, keep the same two send sizes, and run it again. The click rates are 3.707% and 3.913%, a relative lift of 5.6%, with z of 0.73 and a p-value of 0.466. No verdict at all. So subject line B got more people to open the email and there is no evidence it got more people to do anything.

The click-to-open column on the sheet says why: 18.5% against 17.9%, so the extra opens were slightly less likely to click than the original ones, which is what happens when a subject line promises something the email then has to deliver. This is the most common way an email test is misread. Opens are the easiest thing to move and the least valuable thing to have moved; if the campaign exists to produce clicks, the click test is the test, and it needs a bigger send than the open test did because the base rate is five times lower.

Size it before you send with the A/B Test Sample Size tool. What no test can see is the unsubscribes, and a subject line that wins on opens for three months and quietly costs list is a win you pay for later. To use your own send, overwrite the four counts in the panel.

The model

It arrives on a tab called Template: Which Subject Line Won, carrying these columns:

Newsletter send, 18,350 recipients

with the model computed beside the data:

Extra opens: B vs A on full list (count)	328.9
Extra clicks if B is used on the full list	37.81

Once it is in your sheet

The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.

Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.

Next question

Every model like this one, and the method behind them: Statistics in Google Sheets.

Install free All 379 templates