Did the change work, or do your lines just differ?
Ten production lines measured before and after a changeover process was rewritten. The paired test finds a 7.2 unit gain with near certainty, and the unpaired test on exactly the same numbers finds nothing at all.
Words on this sheet
- Production line: One of the parallel lines being compared. Each is a separate machine or crew, so a reading from one is not a reading from the other.
Operations Starter Statistics free
After you install, this is the model to open.
Did the New Process Actually Help?
- In your spreadsheet, click the Sortia icon in the strip of icons down the right-hand edge. No strip? Click the arrow at the bottom-right to open it. You can also use Extensions, then Sortia, then Open Sortia.
- Click Start from a template and put that name in the search box.
- Pick the card with that name and click Load this template. It arrives on a new tab with real numbers already in it.
The answer
- The gain
- 7.2 units an hour, and every one of ten lines improved
- Paired verdict
- p = 0.0000018 t of -10.85 on 9 degrees of freedom
- Unpaired, same numbers
- p = 0.38 no evidence at all; the verdict flips
- Why pairing wins
- 17.5 vs 2.1 spread of the lines against spread of the changes
Ten production lines, each measured for a week before the changeover process was rewritten and a week after. Click Run: the paired test returns t of -10.85 on 9 degrees of freedom with a two-tail p-value of 0.0000018, and the mean gain is 7.2 units an hour. Every line improved, by between 5 and 11 units. Now do the run that makes the point.
Switch to t-Test: Equal Variances and give it the same two ranges. It returns a two-tail p-value of 0.3798, which is no evidence of anything. Same twenty numbers, opposite verdict. The reason sits in the two spread figures beside the data. The lines differ from each other by a standard deviation of 17.5 units an hour, because one of them is old and one is nearly new, while the change on each line has a standard deviation of only 2.1.
The unpaired test has to see a 7.2-unit improvement through a 17.5-unit haze of line-to-line variation and cannot; the paired test never looks at line-to-line variation at all, because it subtracts each line from itself first. If you measured the same thing twice, pairing is not a refinement, it is the difference between finding your result and losing it.
The pairing only works if the rows genuinely correspond, so the first line of the before column and the first line of the after column have to be the same machine, and if the after data arrives sorted differently, sort it back before you run anything. What the test cannot tell you is whether the process caused the gain: ten lines all improving in the same month is also what a new batch of raw material looks like.
To use your own data, put the before figures under Units per hour, before and the after figures beside them under Units per hour, after, one row per line, and widen both ranges.
The model
It arrives on a tab called Template: Did the New Process Help, carrying these columns:
- Line
- Units per hour, before
- Units per hour, after
- Change (units/hour)
with the model computed beside the data:
| Spread across lines (units/hour) | 17.49 |
| Spread of changes (units/hour) | 2.098 |
Once it is in your sheet
- The model arrives with real numbers in it and runs as it stands, so you can press the button first and understand it second.
- Change the numbers to yours. The sheet marks which cells are inputs and which hold formulas, and most labels carry a note explaining the row.
- Press the run button at the bottom of the panel. It is labeled for the tool you are in, and the result lands on its own tab, with a written reading of it beside the figures.
Never used Google Sheets? Start here goes the whole way, in seven steps, and assumes nothing.
Next question
- How many tickets before the event breaks even?Will the Event Make Money?
- What on-time rate can you actually promise?Will We Hit the Delivery Promise?
- The higher price lost. Did it make more money?Two Prices, One Answer
- What should you reserve against a recall?What Would a Recall Cost?
- Book the same room again or take the bigger one?Should We Run It Again Next Year?
- Is a second supplier worth paying for?What If the Single Source Fails?
Every model like this one, and the method behind them: Statistics in Google Sheets.