Did the campaign really move the brand tracker?
Paste the scores from both waves and get back whether the lift is real. Two waves of a brand tracker, sixty respondents each, scored on an instrument whose variance is genuinely known from eleven earlier waves. That is the one condition that makes a z-test the honest tool, and the six-point lift clears easily.
Words on this sheet
- Known variance: The spread you are telling the test to take as given, rather than one worked out from this sample.
- Sample variance: Spread, squared: the standard deviation multiplied by itself.
Marketing Intermediate Statistics free
After you install, this is the model to open.
Two Big Samples, One Real Difference?
- In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
- Click Start from a template and put that name in the search box.
- Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.
The answer
- The move
- +6.0 points of consideration, wave 1 at 61.2 to wave 2 at 67.2
- Real or noise
- p < 0.001 two-tail 0.0000004, with 60 respondents a wave
- Standard errors apart
- 5.1 chance does not plausibly produce this gap
Two waves of the same brand tracker, sixty respondents each. Wave 1 averages 61.23 and Wave 2 averages 67.23, a lift of exactly six points. Click Run: z comes back at minus 5.07 against a critical value of 1.96, with a two-tail p-value of about four in ten million. The lift is real, and the dollars-per-point line beside the data prices it at $40,000 of campaign spend for each point of consideration, which is the number the next budget conversation is actually about.
The reason this is a z-test and not a t-test is the known variance sitting beside the data, and it is the only thing that matters about choosing between them. A z-test needs the population variance to be genuinely known, from outside these two samples, and here it is: the same twelve-item battery has been run eleven times since 2019 and has produced a variance of about 42 every time.
Sortia does not estimate that from your data and will not; you type it in, and if you type in the variance of the samples themselves you have quietly run a worse t-test. Check what happens if the instrument is noisier than you thought. Put 80 into both known-variance boxes and run it again: z falls to minus 3.67 and the two-tail p-value rises from four in ten million to 0.00024.
Still significant, but now visibly resting on a number you asserted rather than measured, and at a variance of 300 the same six-point lift misses the bar entirely. If you cannot defend the variance, use t-Test: Two-Sample Assuming Unequal Variances instead and let the data estimate it. Two things the test cannot see. It does not know whether the two waves asked comparable people, and a tracker that changed panel provider between waves has a bigger problem than any p-value can fix.
And it does not know that the campaign was the cause: six months passed, and everything else that happened in those six months is inside this number too. To use your own tracker, paste one wave per column, put your instrument's historical variance in both boxes, and widen both ranges.
The model
It arrives on a tab called Template: Two Waves, One Real Difference, carrying these columns:
- Respondent
- Wave 1 consideration score (0-100)
- Respondent
- Wave 2 consideration score (0-100)
- Known variance (points squared)
- 42
with the model computed beside the data:
| Dollars per point of lift | 40,000 |
Once it is in your sheet
- The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
- Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
- Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.
Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.
Next question
- Which media split buys you the most orders?Three Media Plans, Compared
- Which client should you turn down?Which Retainers Fit the Team We Have?
- Would three extra days move your critical path?What Sets the Go-Live Date?
- How many retainers can you afford to lose?What If Two Clients Leave at Once?
- Which launch day is a coin toss against the rival, and which is worse?What launch date has even odds of beating the competitor?
- Which budget hits the lead target eight months in ten, and what do the extra points cost?What spend gives 80% odds of hitting the lead target?
Every model like this one, and the method behind them: Statistics in Google Sheets.