Effect size, intervals and "not significant"

What you just saw

Five A/B tests of a conversion rate, each shown as its gap and its 95 % interval. Two lines cut across them. The line at 0 is the test's question: is there any effect at all? The line at 3 points is the business's question: is the effect big enough to act on? Helion's finance team will not fund a rollout for a lift below 3 points, so that is where the second line starts.

The two questions have different answers. A 0.5-point lift measured on a million customers per arm is significant, with an interval from 0.37 to 0.63 points, and it is clearly not worth acting on. A 5-point lift measured on 300 customers per arm is not significant, and it might be worth a great deal: the interval runs from a loss to a large gain. Significance tells you whether chance is a plausible explanation; only the effect and its interval tell you whether it matters.

Significant and important are two different questions

This is the idea that separates a statistically literate analyst from a p-value technician, and it is where most enterprise analytics goes wrong. Because the standard error shrinks with √n, a large enough study will find any non-zero difference significant. A result can be statistically significant and have no practical significance at all.

How large does "large enough" have to be? More than people think. Conversion of 4.0000 % against 4.0004 %, a million visitors per arm, gives z = 0.014 and p = 0.99: nowhere near significant. That gap needs about 52 billion visitors per arm to reach p < 0.001. Likewise 4.00 % against 4.01 % with 500,000 per arm gives p = 0.80, as chapter 3 showed; it needs about 83 million per arm. The point stands with honest numbers: 34.0 % against 34.5 % with a million per arm is overwhelmingly significant, and worth nothing to a business that needs 3 points.

Effect size: how big, on a scale that ignores n

An effect size measures how big a difference is, on a scale that does not depend on the sample size. Each test family has its own:

Cohen's d divides the difference of two means by the standard deviation of single observations, so d = 0.5 means the means are half a standard deviation apart. Cohen's h does the same for two proportions, after a transformation that makes a gap mean the same near 0 %, 50 % and 100 %:

For Helion's Portal v2, 37.97 % of treated customers went paperless against 34.38 % in control, so \(h = 2\arcsin\sqrt{0.3797} - 2\arcsin\sqrt{0.3438} = 0.075\). By the benchmarks that is tiny, far below "small". Yet the raw difference, 3.59 percentage points, is right at the level finance cares about. The raw difference against a business threshold usually beats every benchmark.

The reporting standard: three numbers

Helion's lift: the finding that goes to the board

Helion's paperless lift is significant, p = 4.2 × 10⁻⁵, and its effect size is tiny. Neither number decides anything. The interval does:

Worked: the benefit is customers × saving × lift, so at the interval's lower end \(4\,200\,000 \times 8.40 \times 0.0187 = 0.66\) million euros a year, and at its upper end 1.87 million.

Not significant is not no effect

Before Portal v2, a small pilot tried a new sign-up page on 1,000 customers per arm: 110 converted with the old page and 138 with the new one. The gap is 2.8 points, and the two-proportion z test gives z = 1.90, p = 0.057: not significant at 5 %. Is the new page useless?

The animation says no. It runs the same pilot 1,000 times in a world where the new page really does convert 13.8 % against 11.0 %. The planned power for that gap, with 1,000 per arm, is 0.48, and 473 of the 1,000 pilots found the effect: the other 527 reported "not significant" about an effect that is there. A study with 48 % power fails more often than it succeeds.

So phrase a non-significant result by what it could and could not rule out. Weak: "There is no difference between the pages." Correct: "At the 5 % level there is insufficient evidence that the pages differ." Better still: "…insufficient evidence of a difference. The 95 % interval for the lift runs from −0.1 to 5.7 points, so a lift worth acting on remains consistent with these data." The third version tells the reader what the study could and could not rule out; the first tells them something you did not establish.

The bottles of chapter 6 make the same point. Their 95 % interval runs from 499.87 to 508.13 ml, so the test does not reject 500, but if the tolerance is ±2 ml the interval is alarming: the data are consistent with the line running 8 ml over. The right response is not "no action"; it is "this sample was too small to answer the question: take more bottles".

Can I show there is no difference? An aside on equivalence

A test that fails to reject can never show that two things are the same. An equivalence test can, because it turns the question round: fix, before the data, the largest difference you would still call "no material difference" (the margin), and test whether the true difference lies inside it. The standard version, TOST (two one-sided tests), runs one test against each edge of the margin; equivalence is shown when both reject, which is the same as the 90 % interval lying wholly inside the margin.

Meridian Assurance used it for claim severity. On the log scale the margin was ±0.20, a geometric-mean ratio between 0.82 and 1.22. Telematics claims' geometric mean was 0.930 times the standard tier's, with a 90 % interval from 0.871 to 0.993, inside the margin: TOST p = 0.0007, equivalence shown.

It is tempting to conclude that the tariff can treat severity as unchanged, and Meridian's first readout did. That does not follow. A tariff prices the arithmetic mean claim, not the geometric mean, and the arithmetic means tell another story: telematics claims average 0.793 times the standard tier's, with a 90 % interval from 0.724 to 0.869, well outside the margin. The typical claim is equivalent; the average claim, which the tariff pays, fell by about a fifth. An equivalence test is only as good as the quantity it tests: the estimand must be the one the decision uses. Chapter 20 works through the whole case.

Why it matters

Two decisions go wrong without this chapter. A team with a huge sample ships a change because it is "significant", and finds a year later that the effect was too small to pay for itself. A team with a small sample kills a good idea because it is "not significant", when the study never had the power to see it. Reporting the effect and its interval against the threshold that matters prevents both.

When this goes wrong

Effect-size benchmarks are conventions from one field, not laws: a Cohen's h of 0.075 is "tiny" and worth over a million euros a year to Helion. An interval is only as good as the model behind it: with dependent data, a biased sample or a peeked-at experiment, it is too narrow. And an equivalence test needs its margin fixed in advance, from the decision; a margin chosen after seeing the interval proves nothing.

In code

proportion_effectsize, confint_proportions_2indep and proportions_ztest are in statsmodels.stats.proportion, NormalIndPower in statsmodels.stats.power, and ttost_ind in statsmodels.stats.weightstats. The Excel line for the interval gives its lower end; add instead of subtract for the upper.