Errors and power

What you just saw

A test can be wrong in two ways, and the figure shows both at once. The left curve is the null world: where the average of 25 bottles lands if the line fills 500 ml. The right curve is a world where the line has really drifted to 504 ml. The test does not know which world it is in. It only sees where the average lands, and it rejects H₀ if the average lands beyond the cut-offs.

In the null world, landing beyond the cut-offs is a false alarm; its chance is α, the tails of the left curve. In the drifted world, landing between the cut-offs is a miss, with chance β = 0.484, and landing beyond them is a hit, with chance 0.516: the test's power. With 25 bottles this test catches a real 4 ml drift only about half the time. Move the sliders: more bottles narrow both curves and pull them apart, a bigger drift moves them apart, more noise (σ) widens them, and a looser α moves the cut-offs inwards.

Two ways to be wrong

Because you reason from a sample, you can be wrong in two distinct ways. They are not symmetric, and understanding the difference is what separates someone who can run a test from someone who can design one.

Type I error (α): crying wolf. You announce an effect that does not exist. You recalibrate a machine that was fine, launch a redesign that does nothing, book a benefit that was never there. You control this error directly, by choosing α.

Type II error (β): missing the wolf. A real effect exists and the test failed to detect it. You leave a drifting machine running; you kill a good product. You do not control β directly: it falls out of the sample size, the true size of the effect, the noise in the data and your choice of α.

Power

Power (1 − β) is the probability that a test detects an effect that is really there. It is the most neglected idea in applied statistics. A test with 40 % power is a coin flip dressed up as evidence, and a non-significant result from it tells you almost nothing.

For the z test on the bottles, the power against a real drift \(\delta\) has a formula. In the drifted world z is shifted by \(\delta\sqrt{n}/\sigma\) standard errors, and power is the chance that the shifted z still clears a cut-off:

In words: \(z_{1-\alpha/2}\) is the cut-off (1.96 for α = 0.05), \(\delta\) the drift you want to catch (4 ml), \(\sigma\) the spread of single bottles (10 ml) and \(n\) the number of bottles (25). The shift is \(4 \times \sqrt{25} / 10 = 2\) standard errors, so the power is \(P(Z > 1.96 - 2) = P(Z > -0.04) = 0.516\). The second term, for landing in the wrong tail, is too small to matter here (0.00004).

The four levers

Four things move power. The table starts from the bottles (n = 25, a 4 ml drift, σ = 10 ml, α = 0.05: power 0.516) and changes one at a time.

Sample size is the dominant lever, and the one you can buy. The size of the effect is set by nature, but you choose which effect you care about. Noise can often be cut: a better scale, or measuring the same units twice (pairing, chapter 8). And α trades one error for the other: a looser α catches more drifts and raises more false alarms.

α and power as long-run rates

α and power are promises about the long run, and the animation makes them visible. It runs the bottles' test 1,000 times on a line that is on target: 48 of them reject, close to the 5 % promised. Then it runs it 1,000 times on a line that has drifted by 4 ml: 510 reject, close to the power of 0.516. Any single experiment is simply right or wrong; the rates only show up when you run many.

Choosing α by what each error costs

Lowering α does not make a test better; it moves the balance. For the bottles, α = 0.01 cuts false alarms to 1 in 100 but drops the power against a 4 ml drift from 0.516 to 0.282. So set α by asking what a false alarm costs compared with a miss. A false alarm on the filling line stops production for a needless recalibration; a miss gives product away, or short-changes customers, until the next check. If misses are the expensive error, the answer is a looser α and, above all, a bigger sample.

The two questions to ask about power

• Before the study: how many observations do I need for an 80 % chance of detecting the smallest effect I would actually act on? That is a sample size calculation, and skipping it is how underpowered studies happen.
• After a non-significant result: what is the smallest effect this design could reliably have detected, the minimum detectable effect? Without that number, "we found no difference" cannot be interpreted. With it, you can say the useful thing: "we found no difference, and we would have caught any drift larger than 5.6 ml".

For the bottles with σ known, both answers come from turning the power formula round (leaving out the tiny wrong-tail term):

For 80 % power, \(z_{1-\beta} = 0.8416\). To catch a 4 ml drift: \(n = ((1.96 + 0.8416) \times 10 / 4)^2 = 49.06\), so 50 bottles, always rounding up. With the 25 bottles we have, the smallest drift caught 80 % of the time is \((1.96 + 0.8416) \times 10 / 5 = 5.60\) ml. (With σ estimated from the sample rather than known, the t test needs a little more: 52 bottles, as chapter 6 shows.)

Why it matters

Power decides whether an experiment is worth running at all. A study with 50 % power for the effect you care about will, half the time, report "no significant difference" on a real effect, and someone will read that as "no effect". Planning n before the data, for the smallest effect worth acting on, is the cheapest fix in statistics.

When this goes wrong

Power is worked out for an effect you choose, not the one that exists, which nobody knows. Plan it for the smallest effect worth acting on. Never work it out after the test from the effect you happened to observe ("observed power"): that number is the p-value in disguise and adds no evidence (chapter 17). And the formula here assumes σ is known and the bottles are independent; with σ estimated, the t test's power is a little lower.

In code

norm is scipy.stats.norm; NormalIndPower is in statsmodels.stats.power, and its effect size is the drift in units of σ: 4 / 10 = 0.4. The Excel line for statsmodels leaves out the wrong-tail term, which is too small to change the answer here.