One mean: z and t, one complete test

What you just saw

Chapter 3 knew σ. The line's history said single bottles vary with σ = 10 ml, so the 25 bottles' average of 504 ml gave z = 2.00 and a p-value of 0.0455: reject at 5 %. Now suppose σ is not known and we estimate it from the same 25 bottles, which give s = 10 ml. The statistic is the same number, 2.00, but it is read on a different curve.

On the left, the standard normal: the tails beyond ±2.00 hold 0.0455 of the area. On the right, the t distribution with 24 degrees of freedom: a little flatter, with heavier tails, which hold 0.0569. Below 0.05 on the left, above it on the right: the same data reject in one picture and fail to reject in the other. Drag the degrees of freedom up and the t curve closes in on the normal; drag them down to 4 and its tails swell, to a p-value of 0.1161.

Why σ is never known, and what that changes

Do you know the population standard deviation, σ? Almost never, in practice. If σ is genuinely known (rare: a long-established process with years of history), use a z test. If you estimate it from the sample with s, use a t test.

The t distribution is a slightly flatter bell with heavier tails than the normal, which is exactly the correction needed for the extra uncertainty of having estimated σ. As n grows, s settles on σ, the tails thin, and the t curve converges on the normal: by about n = 30 they are close enough that the distinction rarely changes a decision. Software defaults to t, and that default is correct. Here is how far the bar moves:

The Probability Lab builds the t distribution from scratch in its chapter 11, if you want to see where it comes from.

The one-sample t test

The one-sample t test compares one sample's mean with a hypothesised value. Its statistic is chapter 3's z with s in place of σ:

In words: \(\bar{x}\) is the sample's average, \(\mu_0\) the value H₀ claims, \(s\) the sample's standard deviation and \(n\) the number of observations. The bottom, \(s/\sqrt{n}\), is the estimated standard error of the mean. For the bottles:

Use it when you have one group and a benchmark: is average handling time different from the 4-minute service level? Is the mean fill still 500 ml? It assumes three things:

• Independent observations. One bottle's fill tells you nothing about the next. This comes from how the sample was taken, not from the data.
• A sampling distribution of the mean close to normal. It is, at any n, if the fills themselves are normal. If they are not, the central limit theorem makes the mean's distribution approach the normal as n grows. "n ≥ 30" is a common rule of thumb for that, and only a rule of thumb: the theorem is about the mean, not the data, and strongly skewed data can need far more than 30.
• No extreme outliers. One mis-filled bottle at 400 ml would dominate both x̄ and s at n = 25. Always plot the data before testing it.

Which test? Four questions

Choosing a test is a procedure, not a guess. Four questions, asked in order, settle nearly every case (chapter 15 draws them as one map). For the bottles they go like this:

One complete test, in twelve steps

Here is everything from chapters 1 to 5 applied end to end to one small problem, with every number shown. A manufacturer claims its bottles hold 500 ml on average. An inspector takes a random sample of 25 bottles and finds x̄ = 504 ml with s = 10 ml. Is there evidence, at the 5 % significance level, that the true mean fill differs from 500 ml? Press Step to take the twelve steps one at a time; each puts its number on the same chart.

Steps 1 to 3. The claim concerns the average fill of all bottles, so the parameter is μ, the population mean. Not x̄: we know x̄, it is 504. The manufacturer's claim is the definite value, so it becomes the null, H₀: μ = 500. We have no reason to expect drift in one direction rather than the other, so the alternative is two-sided, Hₐ: μ ≠ 500. And α = 0.05, chosen before looking at the data: we accept a 5 % chance of concluding that the machine has drifted when it has not.

Steps 4 and 5. The four questions give a one-sample t test. Its assumptions: the bottles were sampled at random, so independence comes from the design. Normality deserves a look at n = 25: fills from a calibrated machine are usually close to normal, and a histogram of the 25 would confirm it; badly skewed data at this size would push us to a rank test (chapter 14). And no bottle is wildly off.

Step 6. df = n − 1 = 24. Once x̄ is fixed, only 24 of the 25 deviations from it are free to vary; the last is set by the rule that they add up to zero. Fewer degrees of freedom mean heavier tails and a higher bar.

Step 7. t = 4 / 2 = 2.00. The standard error of 2 ml is the key number: it is roughly how far the mean of 25 bottles strays from μ by chance alone. A 4 ml gap is twice that: suggestive but, as the next step shows, not quite decisive at this sample size.

Steps 8 and 9. The p-value is the chance, if H₀ is true, of a t at least 2.00 from 0 in either direction, on 24 degrees of freedom:

If the machine really were set to 500 ml, samples of 25 bottles would land at least this far from 500 about 5.7 % of the time: unusual, but not unusual enough by the standard we set. The critical values say the same: with 2.5 % in each tail they are ±2.064, and 2.00 falls short of 2.064, so t is not in the rejection region.

Step 10. Both routes agree, because they are the same computation: fail to reject H₀. In the language of the problem: at the 5 % significance level there is insufficient evidence to conclude that the mean fill differs from 500 ml (t = 2.00, df = 24, p = 0.057). Note what is not claimed. We have not shown that the mean is 500 ml, nor that the machine is calibrated; only that these 25 bottles are not, by our agreed standard, strong enough evidence to conclude otherwise.

Step 11. The interval, which you should always report: 504 ± 2.064 × 2, from 499.87 to 508.13 ml. It contains 500, as the test said (chapter 5). But it carries what the verdict throws away: the true mean could be anywhere from just under 500 to over 508 ml. If the tolerance on this product is ±2 ml, that is alarming whatever the p-value. The right response is not "no action". It is "this sample was too small to answer the question: take more bottles".

Step 12. The effect size and the power. Cohen's d expresses the gap in standard deviations, so it does not depend on the sample size:

A d of 0.40 is small to medium by the usual benchmarks: not negligible. How many bottles does a test need to catch a drift like that? That question belongs to the plan, before any bottle is weighed. Say the plant had decided in advance that a 4 ml drift, 0.40 standard deviations, is the smallest worth catching. Then 25 bottles give a power of only 0.484: a coin toss. For the conventional 80 %, the power calculation returns 51.009 bottles, and a sample size is always rounded up: 52 bottles. At 51 the power would be 0.7999, just short.

What would have changed the answer?

Three variations on the same data, each defensible on its own:

This table is uncomfortable, and it should be. The same 25 bottles support opposite conclusions depending on the sample size, the tail and α. That is why those choices must be made, and written down, before the data are collected. Choosing them afterwards, to land on the answer you wanted, is p-hacking, and it is usually committed in good faith by someone who does not realise that is what they are doing. Chapter 18 builds the guardrails.

The full recipe, in one place

• Identify the parameter: μ, p, σ, or a relationship.
• State H₀ and Hₐ, including the tail, before seeing the data.
• Choose α from the cost of a false alarm against the cost of a missed effect.
• Select the test: the outcome, the number of groups, paired or not, σ known or not.
• Check the assumptions, and write down what you would do if they failed.
• Compute the test statistic: signal over noise.
• Find the p-value or the critical value from the right curve and degrees of freedom.
• Decide: reject if p < α, otherwise fail to reject.
• Report the effect size and the confidence interval: how big, and how precisely known.
• Write the conclusion in the language of the problem: not "reject H₀", but what it means for bottles, customers or euros.

Why it matters

The twelve steps turn a test into a decision someone else can check: what was tested, against what, at what risk, with what result and how precisely. For the bottles they change the answer from "the line is fine" to "weigh 52 bottles before deciding", which is the difference between shipping an overfill of up to 8 ml unnoticed and catching it.

When this goes wrong

The t test is about the mean, and in small samples it is fragile. One bottle at 400 ml among 25 would pull x̄ down by 4 ml and inflate s, and could turn any verdict. With 25 fills that are close to normal the test is safe; with 10 values from a long-tailed process it is not, and a rank test (chapter 14) is the honest choice. No test can rescue a sample that was not random either: bottles taken from the top of one pallet, or all from one shift, can mislead with any p-value.

In code

In Excel, B1 holds t. t is scipy.stats.t and TTestPower is in statsmodels.stats.power. Excel has no non-central t, so its power formula uses the normal curve and gives 49.1 bottles, a little low: take the exact 51.009 from Python and round up to 52.