Tests and confidence intervals

What you just saw

Slide x̄ and watch two things at once: the interval around it, and the verdict of the test of 500 ml. They change together. The moment 500 slips out of the interval, p drops below 0.05 and the test rejects; slide back, and both flip back. They are not two methods that happen to agree. They are one calculation, read two ways.

An interval instead of a verdict

A test returns a verdict. A confidence interval returns a range of parameter values consistent with your data: strictly more information, and in the units the business already thinks in.

The Probability Lab builds intervals from scratch in its chapter 11; here is the short version. Every interval has the same anatomy: the estimate, plus or minus a critical value times the standard error.

In words: \(\bar{x}\) is the estimate, \(\sigma/\sqrt{n}\) (or \(s/\sqrt{n}\)) its standard error, and \(z^{*}\) or \(t^{*}\) the critical value that leaves 2.5 % in each tail for a 95 % interval: the same critical value the two-tailed test uses.

The bottles' two intervals

With σ = 10 ml known (chapter 3), \(z^{*} = 1.96\) for 95 % and the standard error is 2 ml:

500 is outside, only just, and chapter 3's z test rejected. Now suppose σ is not known and we use the sample's s = 10 ml instead, as the tutorial's worked example does. The critical value then comes from the t curve with n − 1 = 24 degrees of freedom, \(t^{*} = 2.064\), which is wider than 1.96 because an estimated σ adds uncertainty:

Now 500 is inside, and the t test (chapter 6) fails to reject. Same bottles, a different assumption about σ, and the interval and the test flip together.

The duality, and where it stops

For a mean tested with z or t, the two cannot disagree: the test and the interval use the same standard error and the same critical value, so they are the same computation viewed from two directions. That makes the interval a free check on your arithmetic.

It is not true of every test. For a proportion, the usual z test measures the gap with the standard error that the null value implies, \(\sqrt{p_0(1-p_0)/n}\), while the usual (Wald) interval uses the one the data imply, \(\sqrt{\hat{p}(1-\hat{p})/n}\). Near the edge they can disagree. Say a supplier promises that at most 10 % of parcels arrive damaged, and 6 of 25 do (24 %). The z test of p = 0.10 gives z = 2.33 and p = 0.0196, so it rejects; yet the 95 % Wald interval, 7.26 % to 40.74 %, contains 10 %. Chapter 10 shows the same thing for two proportions and says which interval matches which test.

Why report the interval

Prefer intervals in reporting. "The mean fill is 504 ml, 95 % CI 499.87 to 508.13 ml" tells a plant manager the plausible range and whether it clears their tolerance. "p = 0.057" tells them only that 500 is not ruled out, which is often the least interesting thing they wanted to know.

The interval also carries information the verdict throws away. It says the true mean could be anywhere from just under 500 to over 508 ml. If the tolerance on this product is ±2 ml, that is alarming whatever the p-value: the data are entirely consistent with the line running 8 ml over. The right response is not "no action". It is "this sample was too small to answer the question: take more bottles".

What the 95 % promises

The 95 % belongs to the procedure, not to any one interval: of all the intervals built this way, 95 % contain the true mean. The animation draws 100 samples of 25 bottles from a line truly at 500 ml. 95 of the intervals contain 500 and 5 miss it, and each miss is also a t test of H₀: μ = 500 that rejects a true null. The 5 % of intervals that miss and the 5 % false-alarm rate are the same 5 %.

Where the Probability Lab left off

The Probability Lab ended its chapter 11 with a coffee machine: 50 cups (its dataset coffee-caffeine.csv) with a mean of 113.18 mg of caffeine and s = 8.01 mg, against a maker's claim of 110 mg. The 95 % t interval runs from 110.91 to 115.46 mg. 110 lies outside it, so the t test of H₀: μ = 110 rejects at 5 %: t = 2.81 on 49 degrees of freedom, p = 0.0071. The interval says more than the verdict: the machine pours between about 0.9 and 5.5 mg more caffeine than claimed.

Why it matters

A decision is rarely "is it exactly 500?". It is "is it close enough to leave alone, or far enough to act on?". The interval answers that in millilitres, the unit the decision is taken in, and the test's verdict comes free with it. That is why this lab reports an interval beside every p-value.

When this goes wrong

The duality is exact only when the test and the interval share their standard error and their tails: a two-sided test with a two-sided interval at the same level, or a one-sided test with a one-sided bound. For proportions and counts the usual interval and the usual test are built differently and can disagree near the boundary, as the damaged parcels showed. And the 95 % describes the procedure: any one interval either contains the true mean or it does not.

In code

Excel's CONFIDENCE.NORM and CONFIDENCE.T return the margin of error, not the interval: subtract it from the mean and add it to the mean. proportions_ztest and proportion_confint are in statsmodels.stats.proportion; without prop_var, proportions_ztest uses the sample's proportion in the standard error, so pass the null value to get the textbook test.