Two questions pick your test. The rest is a tiebreak.

The chooser

Read down the first column until a row describes the thing on your screen. The middle column is the name as it is written in the tool list, so you can search for it. The last is a worked example with real numbers, readable whether or not you ever run anything.

The middle column is the tool list of the Sortia sidebar, verbatim, down to the three names that carry (nonparametric) in the picker itself. That is the list telling you the same thing this page does: those three make no assumption about the shape of your numbers. Two of them have a floor on how little data they will accept: the signed-rank test refuses fewer than five pairs that actually changed, and Mann-Whitney refuses fewer than three values on either side.

Two honest gaps in that table. There is no one-sample t-test on this bench. For a single column against a fixed target, the half-width that Descriptive Statistics prints on its Confidence Level row answers the same question, once you do the last step yourself: add it to the mean and subtract it, and if your target sits outside that pair of numbers, the gap is larger than samples that size usually produce by chance at that level. And there is no repeated-measures ANOVA and no mixed model anywhere, so a design with three or more measurements on the same subjects is past what this list covers. Reducing it to one difference per subject and running a paired test answers a narrower question honestly, which is better than running a wider test that is not there.

When the disagreement is total

Eight page loads served by an old content network and eight by a new one. Different pages, not the same eight measured twice, so the rows are independent and the shortlist is the two-sample tests. Each column happens to carry one multi-second load, which is what real latency data looks like.

Two tests, both legitimate for this design, on those sixteen numbers:

The equal-variances t-test returns exactly the same figures here, t 0.359 and p 0.7253, so this gap is not about which t-test you pick. Both columns are above; in the add-on they are the model named Faster Site, Proven Without the Bell Curve. Computed 6 September 2026.

One test says 0.0104 and the other says 0.7253. That is not a rounding disagreement. One of them says the new network is faster and the other says there is nothing here to see.

The two slow loads are doing all of it. A t-test measures the gap between the averages against the spread inside the groups, and 2,900 ms and 3,100 ms push those variances to roughly 760,000 and 1,040,000. Against a spread that size a 170 ms gap between the means is nothing, which is exactly what a p of 0.73 is reporting. The rank test never sees the number 3,100 at all. It records only that this was the slowest of the sixteen loads, worth one rank like every other value, and then asks whether the new network’s loads tend to sit lower in the pooled order. Compare every old load against every new one, all 64 pairings, and the new network is the faster of the two in 56 of them. The 8 pairings that go the other way are the U of 8 in the table above.

So report the medians. 450 ms against 217.5 ms, roughly twice as fast for a typical visitor. The means, 746.9 and 576.9, describe neither network’s behavior on any real page: no load in either column is anywhere near them. Once you have chosen a rank test because the extremes were distorting the averages, quoting those same averages in the summary undoes the choice.

And notice the row that says how the p was computed. Here it says exact: no two values in the pooled sixteen are tied and the groups are small enough to enumerate, so every possible arrangement was counted rather than approximated. In the ticket example above, the same row said normal approximation, because ties among the differences made counting impossible. The same tool, the same row, two different answers about its own arithmetic. Any tool that prints a p-value without telling you how it got there is asking for more trust than it has earned.

Choosing a rank test is not the same as throwing the outliers away. A 2.9-second load is a real load and a real person waiting. The rank test keeps that row in the data and refuses to let it set the verdict, which is a very different act from deleting it. If the slow tail is the thing you actually care about, then the extremes are the finding rather than the noise, and the instruments are a look at the outliers and a histogram with the cumulative column, not a test of the middle.

What a result has to carry

A p-value on its own is not a result. Nobody can check it, nobody can tell whether the effect is worth acting on, and nobody can tell which test produced it. Whatever tool you use, these are the things to insist on beside it, and every one of them appears in the tables above:

• The size of the effect, in the units of your problem. A median cut of 4 hours per ticket. 15.3 minutes an invoice. 450 milliseconds against 217.5. That is the number the decision is made on; the p-value only says whether it is worth believing.
• The middle of each group, computed the way the test computes it. Means for a t-test, medians for a rank test. Quoting the means beside a rank test undoes the reason you chose it.
• The spread, and how many rows. Two groups with the same averages and hundredfold different variances are not the same finding, and the variance rows are where that shows up.
• The degrees of freedom. The invoice example is two tests with the same statistic separated by nothing else, and it moved the p-value more than tenfold.
• How the p-value was computed. The two rank tests above print it: exact when every arrangement was counted, normal approximation when ties or size made that impossible. Kruskal-Wallis prints no such row, and nothing else here does either. It is a rare row and it deserves to be common.

Sortia writes all of those into a tab of your own spreadsheet, as a table, with a panel beside it saying in plain words what the figures mean, so the artifact you forward is checkable by whoever receives it rather than a screenshot of one number. One display note worth knowing: in the report tab the t-test, z-test and F-test tables format their value columns to three decimals, so a p-value below 0.0005 reads as 0.000 in the cell. The full value is still there and the formula bar shows it when you click the cell. The reading beside the table will not: that sentence writes anything under 0.001 as “well under 0.001” rather than printing the figure. The panel itself goes to the right of the table on a fresh report tab; append a run into a tab of your own and the numbers land without it.

What this page does not settle

• Whether your data is normally distributed. Nothing in the tool list returns a verdict on that. There is no Shapiro-Wilk and no Levene. Distribution Fitting comes closest, and what it does is narrower than a verdict. It ranks candidate shapes against each other by Kolmogorov-Smirnov distance, the normal curve among them, and prints an Anderson-Darling statistic beside each one. It does carry two p-value columns beside the candidates, one from a chi-square and one from the K-S distance, and both headings call the figure an approximation, because the parameters were fitted from the same data those p-values are then judged against. The Anderson-Darling column deliberately carries no p-value at all, for the same reason taken to its conclusion: once the parameters were fitted from the same data no honest generic one exists. The report’s own note says to read a small p as evidence against a family and not to read a large one as proof of it, which is a ranking with a caveat on it rather than an answer to whether your data is normal. The absence is real, and it is less serious than it sounds: the shape question matters far less for comparing two averages than most people have been told, and far more for a tail percentile. Why the shape question is usually the wrong one takes that apart.
• Which group differs, once three or more of them do. A significant F says at least one group is unlike the others and never which, and there is no post-hoc test here. ANOVA in Google Sheets does the follow-up arithmetic out loud, including why the threshold has to move when you run the pairs.
• Which of the three t-tests fits your design, in more detail than one row of the chooser. t-Test in Google Sheets works through each of them, and through the built-in formula beside them.
• Anything with repeated measures or mixed effects. Three or more measurements on the same subjects, or subjects nested inside groups, is past what this bench covers. Saying so is more useful than bending a test that is here into a shape it does not fit.

And it does not settle the biggest question of all, because nothing can: there is no single correct test for a situation. Three of the sections above run two defensible tests on one dataset and get different answers back, and in one of the three the difference changes the verdict. What exists instead is a test that matches the design you actually ran, a set of assumptions you have to be willing to say out loud, and an effect size that has to be worth acting on before the p-value is even interesting.

If you ran two and they disagreed, the useful move is not to quietly keep the smaller p. It is to say which one you chose before you looked, and why, and to report the other one beside it.