The lift is real. Is it real for every segment?
A significant 22.4% lift in activation that comes apart the moment you split it. All of it is in team signups and none of it is in self-serve, which changes what you ship next.
SaaS Intermediate Statistics free
After you install, this is the model to open.
Did the New Onboarding Move Activation?
- In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
- Click Start from a template and put that name in the search box.
- Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.
The answer
- The headline
- +22.4% activation lift, 23.9% to 29.2%, p = 0.032
- Self-serve alone
- p = 0.61 no evidence of any effect
- Team signups alone
- +32.9% 21.4% to 28.5%, p = 0.019
- Ship it to
- teams the whole lift lives in one segment
Three weeks of signups, 620 on the current onboarding flow and 640 on the new one. This is the one tool in the panel that takes no range at all, just four counts, so the sheet is the working and the panel is the input. Click Run on the counts already there: activation goes from 23.87% to 29.22%, an absolute lift of 5.35 percentage points and a relative lift of 22.4%, with z of 2.15 and a two-tail p-value of 0.0317.
Significant, shippable, and the deck writes itself. Now do the two runs that decide what you actually ship. Self-serve signups first: type 61 of 214 against 64 of 208. That is 28.50% against 30.77%, z of 0.51 and a p-value of 0.610. Nothing. Then team signups: 87 of 406 against 123 of 432. That is 21.43% against 28.47%, a relative lift of 32.9%, z of 2.35 and a p-value of 0.0187.
The entire headline result lives in one segment. The new flow works for people arriving with colleagues and there is no evidence it does anything at all for people arriving alone. That is a different product decision from the one the headline supports: ship it to team signups, keep looking for the self-serve problem, and do not spend the next quarter defending a number that was never about self-serve.
Two cautions, because segment-splitting is also the easiest way to fool yourself. Decide your segments before you run the test rather than after you see the result, because with enough cuts something always looks significant. And notice that the self-serve run is not evidence of no effect: 214 signups an arm can only detect a very large difference, and a real eight percent lift there would be invisible at this sample size.
Size the segment you care about with the A/B Test Sample Size tool before the next test rather than after it. The two lines at the foot of the sheet price the decision that follows. At 432 team signups every three weeks, a 7.04 point lift is about 517 extra activated users a year, worth about $486,000 at $940 each, and that is the number to put beside the cost of finishing the rollout.
Enter counts and never rates: the test works from the number of people behind each percentage. What no split can fix is a three-week window on a product where activation takes ten days, so some of the late signups in both arms had not finished being measured when the test closed. To use your own test, overwrite the four counts in the panel for each cut in turn.
The model
It arrives on a tab called Template: Did the New Onboarding Work, carrying these columns:
- Activation test, 1,260 signups over three weeks
with the model computed beside the data:
| Extra activations a year, team rollout only | 517.3 |
| Value of those extra activations ($ a year) | 486,248.6 |
Once it is in your sheet
- The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
- Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
- Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.
Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.
Next question
- What does hiring now cost you next January?What Three Hiring Plans Cost
- Your traffic is fixed. What lift can it even see?What Lift Could We Even Detect?
- Do the service credits or the churn cost you more?What Would an Outage Cost Us?
- Commit to the servers or rent them by the hour?Which Instance Mix Holds the Load for the Least Money?
- Is the migration in trouble, or just early?Buffer the Migration, Not the Tasks
- Is it cheaper to staff the night or to buy it in?Cover the Support Hours for the Least Money
Every model like this one, and the method behind them: Statistics in Google Sheets.