Two independent means

What you just saw

Helion Energy, a European energy retailer, rebuilt its self-service billing portal and randomised 12,000 customers between the old portal and the new one, Portal v2, for one billing cycle. One of finance's questions: do customers on Portal v2 pay their bills sooner? Days from bill to payment tie up working capital, so a day matters.

The animation let the customers in a few at a time, in random order. With 4 per group the difference in mean days to payment was −1.58 days, but the curve showing how much such a difference wobbles was so wide that it straddled zero comfortably. At 15 per group the difference even pointed the wrong way, +0.26 days. That is what small samples do. As the groups grew, the curve narrowed, moved clear of zero and settled on the answer from all 12,000: Portal v2's customers paid 1.09 days sooner, with a 95 % interval from −1.21 to −0.97 days.

Two things moved the verdict: the size of the gap, which barely changed once the groups had a few hundred customers each, and the width of the curve, which kept shrinking as n grew. The test statistic is their ratio.

The two-sample t statistic

The two-sample t test compares the means of two separate, unrelated groups: control against treatment, region A against region B. Its statistic has the same shape as every other: the signal, the gap between the two sample means, over the noise, the standard error of that gap.

In words: \(\bar{x}_1\) and \(\bar{x}_2\) are the two groups' means, \(s_1\) and \(s_2\) their standard deviations, and \(n_1\) and \(n_2\) their sizes. Each group's mean wobbles by \(s/\sqrt{n}\); the gap between two independent means wobbles by the square root of the sum of their squares. For Helion, with Portal v2 as group 1 (6,031 customers, mean 14.355 days, s = 3.40) and the old portal as group 2 (5,969 customers, mean 15.445 days, s = 3.48):

The gap is more than 17 standard errors from zero. If the portal made no difference, a gap like this would essentially never happen: p = 1.3 × 10⁻⁶⁶.

Welch's t test, by default

The formula above is Welch's t test. The classic Student version differs in one respect: it assumes the two groups have the same variance, pools the two standard deviations into one, and uses n₁ + n₂ − 2 degrees of freedom:

For Helion the choice hardly matters: the pooled test also gives t = −17.35, because the groups are nearly the same size and their spreads are close. It matters when a small group has the larger spread. The animation below runs 1,000 experiments in which the two populations have the same mean, so every rejection is a false alarm: 10 values with a standard deviation of 20 against 40 values with a standard deviation of 5.

The pooled test rejected 311 times in 1,000, a false-alarm rate of 31.1 % where it promised 5 %: pooling lets the 40 calm values speak for the 10 wild ones, so its standard error comes out far too small. Welch's test rejected 48 times, 4.8 %, as promised.

The assumptions, and what the checks decide

The two-sample t test assumes independence within and between the groups (one row per customer; randomisation takes care of the rest), approximately normal sampling distributions of the two means, and, for the pooled version only, equal variances. Helion's analysis checks each before running the test:

Welch's test is used regardless of Levene's result: it is valid whether the variances are equal or not, and costs almost nothing when they are. Running Levene first and switching to the pooled test when it does not reject is exactly the habit the callout above rules out.

The result, and how big it is

The effect size for two groups is Cohen's d: the gap divided by the pooled standard deviation, 3.44 days, so d = −0.32. Here the effect is both statistically overwhelming and practically meaningful: about a day off the working-capital cycle, with an interval tight enough that the size is not in doubt.

The Mann–Whitney test is there because a reviewer will ask whether the mean is the right summary for skewed billing data. It compares the two groups without assuming normality (chapter 14), and it rejects too, in the same direction: Portal v2's customers tend to pay sooner. Being able to say that both tests agree closes that line of challenge before it opens, and the lab says it only because both results say so.

Why it matters

A retailer with millions of customers counts working capital in days: paying a day sooner across the book is real money, and the interval says the gain is at least 0.97 days. Choosing Welch by default means the decision never rests on an assumption about equal spreads that nobody checked, and the robustness check means it does not rest on the mean alone.

When this goes wrong

Independence is the assumption no test can check. If the same customer appeared twice, or if customers were randomised in households or regions rather than one by one, the effective sample would be smaller than the row count and every standard error too small. In small samples of skewed data the t test's normal approximation fails too, and the mean may not be the question anyway: chapter 14's rank and permutation tests are the alternatives. And a pooled test with a small, noisy group can raise six times the false alarms it promises, as the animation showed.

In code

ttest_ind, levene and mannwhitneyu are in scipy.stats. Excel's T.TEST returns the p-value only: its last argument 3 is Welch, 2 is pooled. Excel has no Levene or Mann–Whitney test: W comes from an ANOVA on the absolute deviations from each group's median, and the Mann–Whitney route starts from the ranks (chapter 14).