The p-value

What you just saw

Back to the bottles, now with one more fact: the line's long history says single bottles vary with σ = 10 ml, so we take σ as known. The 25 bottles averaged 504 ml. The curve is the null world for the test statistic z: how z would vary from sample to sample if the machine really filled 500 ml.

The bottles' z of 2.00 sits on it, and the shaded area beyond ±2.00 is 0.0455: the p-value. Drag the marker. The nearer z is to 0, the bigger the shaded area, and the less surprising the data would be if nothing had changed.

The test statistic: signal over noise

Here is the idea behind every test in this lab. Every test statistic has the same shape:

The top is the signal: how far the data landed from the null's prediction. The bottom is the noise: the standard error, how much a statistic like this varies from sample to sample when nothing real is happening. The ratio turns a gap in millilitres, euros or days into a count of standard errors with no units, which is what lets one reference curve serve every problem.

For a mean with σ known, the recipe reads:

In words: \(\bar{x}\) is the sample's average (504 ml), \(\mu_0\) the value H₀ claims (500 ml), \(\sigma\) the known spread of single bottles (10 ml) and \(n\) the number of bottles (25). The bottom, \(\sigma/\sqrt{n}\), is the standard error of the mean. For the bottles:

The sample mean landed two standard errors above where H₀ said it would. That is what a z of 2.00 means, whatever the units. Every test in this lab follows the same skeleton; only the signal and the noise change:

Why the noise shrinks as n grows

The standard error of a mean is \(\sigma/\sqrt{n}\), so it shrinks as n grows, and the same gap gives a bigger z. That is not a trick; it is right. A 4 ml gap in 4 bottles is weak evidence (z = 0.80). In 25 it is suggestive (z = 2.00). In 25,000 it is overwhelming (z = 63.25), because the average of 25,000 bottles simply cannot wander 4 ml off target by chance.

It has a consequence that runs through the whole lab: with a large enough sample, any difference, however trivial, becomes statistically significant. Significance is not importance (chapter 16).

The p-value

Everything hangs on the last clause. The p-value is worked out inside the null world. It answers: if nothing were really going on, how often would data like mine turn up?

• Small p: data like mine would be rare if H₀ were true. Either something unusual happened, or H₀ is wrong. That is evidence against H₀.
• Large p: data like mine are unremarkable if H₀ is true. There is no reason to abandon it.

For the bottles the test is two-tailed, so "at least as extreme" means at least 2.00 standard errors away, in either direction:

If the machine really filled 500 ml, a sample of 25 bottles would land at least 4 ml from 500, one way or the other, about 4.6 % of the time.

The decision rule

Before collecting any data you choose a threshold for how much evidence you require: the significance level, α. Then the rule is mechanical:

With α = 0.05, p = 0.001 rejects, p = 0.02 rejects and p = 0.20 does not. The bottles' 0.0455 is below 0.05, so, with σ known, we reject H₀: the evidence says the machine has drifted. Small or not is judged only against the α you fixed in advance: a p-value of 0.0985 is not "very small" at α = 0.05; it simply fails to reject.

α = 0.05 means you accept a 5 % chance of rejecting H₀ when it is in fact true. It states your tolerance for false alarms; it is a decision about consequences, not a mathematical fact. There is nothing sacred about 0.05: it is a convention inherited from the statistician Ronald Fisher and hardened by habit. Set it by asking what a false alarm costs compared with a missed effect:

• Screening thousands of compounds for a follow-up assay: a false positive costs a cheap re-test, a false negative loses a drug. Use a loose α (0.10).
• Approving a treatment for patients: a false positive harms people. Use a strict α (0.01 or lower).
• A/B testing a button colour: either error costs almost nothing. 0.05 is fine.
• Particle physics: the standard is five sigma, about α = 0.0000003, because the claim is extraordinary and the data are enormous.

Critical values: the same decision from the other side

There is a second route to the same decision. Instead of the area beyond z, mark off the most extreme α of the null curve in advance. That is the rejection region, and its edge is the critical value. For a two-tailed z test at α = 0.05 the curve splits into 2.5 % in the left tail, the middle 95 % and 2.5 % in the right tail, so the critical values are ±1.96. The bottles' z = 2.00 lies beyond 1.96, inside the region: reject, as before.

Switch the figure at the top to the rejection region and drag the marker: it enters the region at exactly the point where the p-value drops below 0.05. The two routes always agree, because they read the same curve.

Before computers the critical value was the only practical route; it survives because it makes the geometry of the test visible. Chapter 6 uses it with the t curve, whose critical values for 25 bottles are ±2.064 rather than ±1.96.

What a p-value is not

Four misreadings, all common and all wrong. Learning to spot them is worth more than any formula in this lab. Turn each card over to see why.

The arbitrariness of 0.05

There is no meaningful difference in evidence between p = 0.049 and p = 0.051, yet the first is often called a discovery and the second a null result. Treat 0.05 as a rough guide rather than a cliff edge, report the actual value, and let the effect and its confidence interval carry the message.

"Fail to reject": why the wording matters

When p ≥ α, the correct phrase is "we fail to reject H₀". Not "we accept H₀", and certainly not "we proved there is no difference". The difference is real.

A large p-value has two possible causes, and the test cannot tell them apart: either there really is no effect, or there is one and the study was too small, too noisy or too short to see it. Since you cannot tell which, you cannot claim the first. The court reaches "not guilty", not "innocent".

The third version tells the reader what the study could and could not rule out; the first tells them something you did not establish. The example is the bottles again, with σ estimated from the sample instead of known: chapter 6 shows why the same 25 bottles then fail to reject.

Why it matters

A p-value is how a test turns "that looks off" into a decision you can defend: recalibrate the filling machine or leave it, ship the page or not. Fixing α before the data and reading p against it means the decision does not bend towards what you hoped to see. Reporting the p-value itself, not just "significant", lets the next person apply their own threshold.

When this goes wrong

A p-value is only as good as the null world it comes from. The z test assumes that σ really is known and that the bottles are independent; if σ was guessed too low, z comes out too big and p too small. A p-value also depends on n as much as on the effect: the same 4 ml gap gives p = 0.0455 with 25 bottles and p = 0.4237 with 4. And it answers one narrow question, how surprising the data are if H₀ holds. It says nothing about how likely H₀ is, or how big the effect is.

In code

In Excel, B1 holds z. norm is scipy.stats.norm. Beware Excel's Z.TEST(A2:A26, 500, 10): it returns the right-tailed p-value, not the two-tailed one, a well-known trap. For a two-tailed test, use the formula above.