Power and sample size

What you just saw

Helion is planning its next experiment on paperless billing. The baseline is 34 %, and finance will not fund a change that lifts it by less than 3 points. How many customers per arm does the test need to catch a 3-point lift 80 % of the time? The curve answers: power rises with every customer added, slowly at first, and crosses 80 % at 3,993 customers per arm.

The two worlds above the curve show why. They are the test statistic's distribution if there is no lift and if the lift is real. Every extra customer moves the second world further from the first, so more of it lands beyond the test's cut-offs. Move the lift slider: a smaller lift pushes the curve to the right and the sample size up, fast; halving the lift roughly quadruples the sample.

The two questions to ask about power

• Before the study: how many observations do I need for an 80 % chance of detecting the smallest effect I would actually act on? That is a sample size calculation, and skipping it is how underpowered studies happen.
• After a non-significant result: what is the smallest effect this design could reliably have detected, the minimum detectable effect? Without that number, "we found no difference" cannot be interpreted. With it, you can say the useful thing: "we found no difference, and we would have caught anything larger than this".

Both questions are asked about the effect worth acting on, which comes from the decision, not from the data. That is the whole difference between planning power and misusing it.

How many customers would we need?

For two proportions, the calculation runs on Cohen's h (chapter 16), and the sample size per arm comes from turning the power formula round:

In words: \(z_{1-\alpha/2}\) is the test's cut-off (1.96 for α = 0.05, two-sided), \(z_{1-\beta}\) the extra distance needed for the power you want (0.8416 for 80 %), and \(h\) the lift you want to catch on Cohen's scale. For Helion, \(h = 2\arcsin\sqrt{0.37} - 2\arcsin\sqrt{0.34} = 0.063\), so \(n = 2 \times ((1.96 + 0.8416) / 0.0627)^2\), just over 3,992; always round up, to 3,993 per arm.

Smaller effects cost more than people expect. A lift from 10 % to 12 % conversion, a common target for a sign-up page, has h = 0.064 and needs about 3,835 customers per group; a 1-point lift from 10 % needs about four times as many.

What is the smallest effect we could have detected?

For days to payment at 5,969 customers per arm, the minimum detectable effect at 80 % power is a Cohen's d of 0.051, or 0.18 days in real units (d times the standard deviation of days to payment, 3.48 days). The observed effect of 1.09 days sits comfortably above it.

This second number is what makes a null result interpretable. "We found no difference" is uninterpretable on its own. "We found no difference, and this design would have detected anything larger than 0.18 days" is a finding. Report the MDE alongside every non-significant result and the file drawer stops being a black hole.

Planning, not observed power: the bottles

The tutorial's bottles (chapter 6) did not reject H₀: μ = 500 with 25 bottles, t = 2.00 and p = 0.057. The worked example then asks how much power the study had. Done right, this is planning: the drift worth catching on the filling line is 4 ml, so with σ about 10 ml the effect is d = 0.40, and the one-sample t test with 25 bottles had power 0.484 against it. To reach 80 %, the solver gives 51.009 bottles, which rounds up to 52 (at 51 the power is 0.7999, just under the target).

Done wrong, the same arithmetic is observed power: power computed at the effect the sample happened to show, after the test. It adds no evidence, because it is a restatement of the p-value: a result exactly on the 5 % line always has an observed power of about 0.50, whatever the study. The 0.484 above is legitimate only because 4 ml is the drift the plant cares about, chosen before looking; it would be meaningless as a verdict on this sample.

Power for the other tests: ANOVA and chi-square

Every test has a power calculation; only the effect size changes. For ANOVA it is Cohen's f, for a chi-square test Cohen's w (the square root of the summed squared differences between the true and the hypothesised shares, each divided by the hypothesised share). Helion's channel-mix test compares 1,600 contacts with the forecast shares over 4 channels; against a small shift in the mix (w = 0.1) it has power 0.93, comfortably above 80 %. It would need 1,091 contacts for exactly 80 %.

Why it matters

An experiment sized by whatever traffic showed up is a coin flip dressed as evidence. Sizing it first, for the smallest effect worth acting on, tells you whether to run it at all, how long it must run, and what a null result will mean. It is the cheapest decision in the whole readout.

When this goes wrong

A power calculation is only as good as its inputs: the baseline rate, the spread and the effect are all guesses before the data, and a wrong baseline moves the answer. Plan for the smallest effect worth acting on, not the effect you hope for; the hoped-for effect is always bigger and gives a sample that is too small. Allow for customers who drop out, and remember that testing several metrics or peeking at the data (chapter 18) changes the error rates the calculation assumed.

In code

The power classes are in statsmodels.stats.power, proportion_effectsize in statsmodels.stats.proportion. Excel has no non-central t, F or χ², so its lines are the normal approximation: close for large samples (the bottles need 49.06 by it, against 51.009 with the t test's exact power), rougher for χ².