Hypotheses and tails

What you just saw

The curve is the null world for the bottles: where the average of 25 bottles lands if the machine really fills 500 ml on average. (With σ = 10 ml, that average varies with a standard error of 2 ml; chapter 3 explains why.) The shaded region is where an average would have to land for us to stop believing the machine is on target.

Which region depends on the question. "Is the fill different from 500?" shades both ends; "is it more than 500?" shades only the right; "is it less?" only the left. Everything in this chapter is about writing that question down precisely, and doing it before the data arrive.

Four words: population, sample, parameter, statistic

Four words are used precisely throughout this lab. Mixing them up is the commonest source of early confusion.

The two hypotheses

A test always sets up two competing statements about a parameter. They cannot both be true, and between them they cover every possibility, so ruling one out leaves only the other.

The null hypothesis, H₀, is the position of no effect, no difference or the claimed value: the status quo, the dull explanation, the thing you assume until the evidence forces you off it. It always contains an equality: =, ≤ or ≥. That is not an arbitrary convention. The null must name a definite value so that you can work out probabilities under it; you cannot say how surprising your data would be under a hypothesis as vague as "the mean is somewhere above 500".

The alternative hypothesis, Hₐ (also written H₁), is what you are gathering evidence for: an effect exists, the groups differ, the claim is wrong. It contains a strict inequality: ≠, > or <.

The courtroom, and where it breaks

The usual picture is a criminal trial. The court starts by assuming the defendant is not guilty (H₀). The prosecution presents evidence. If the evidence passes the standard of proof, the assumption is dropped and the verdict is guilty (reject H₀). If not, the verdict is not guilty, which is not the same as proven innocent.

That last point matters most, and chapter 3 comes back to it. But see where the picture breaks. A court weighs evidence by judgement, while a test works out an exact probability. And a court has one defendant, whereas an analyst runs many tests. Suppose an analyst runs forty tests a quarter on changes that in truth do nothing. At a 5 % threshold, about two of them will come out "guilty" by chance every quarter: eight innocents convicted a year. Chapter 18 deals with that.

Setting them up: the bottles

A maker claims that its bottles contain 500 ml on average. You suspect the filling machine has drifted, but you have no reason to think it drifted in a particular direction. Then:

Look at what went where. The maker's claim, 500 ml, went into the null, because it is the definite value. Your suspicion, drift, went into the alternative, because it is what you need evidence to establish. Beginners often swap them round and then find the test does something they did not expect.

One tail or two

The form of the alternative decides where the evidence has to fall to count against H₀. The figure at the top of the chapter shows the three cases:

• Two-tailed, Hₐ: μ ≠ 500. You care about a departure in either direction: a mean of 480 and one of 520 are both interesting. The rejection region sits in both tails, with α split between them: 2.5 % in each when α = 0.05.
• Right-tailed, Hₐ: μ > 500. You only care about an increase: a process change that raises throughput, a drug that improves outcomes. All 5 % of the region sits in the upper tail.
• Left-tailed, Hₐ: μ < 500. You only care about a decrease: under-filling, a defect rate that has fallen, a load time that has improved.

For the bottles, with a standard error of 2 ml, the average must land below 496.08 or above 503.92 ml to reject two-tailed at 5 %. One-tailed, it need only clear 503.29 ml (or fall below 496.71 ml). A one-tailed test is more powerful in the direction it looks, because it puts the whole of α in one tail.

Choose the tail before you see the data

The animation shows why the choice has to come first. It ran 1,000 experiments in a world where the machine is on target, so every rejection is a false alarm. The honest analyst chose a two-tailed test in advance: 49 false alarms, close to the 5 % promised. The peeker looked at which way the data leaned and then tested that tail alone: 98 false alarms, close to 10 %. Picking the tail after seeing the data quietly doubles the false-alarm rate.

Why it matters

Writing H₀ and Hₐ down, with the tail, before the data exist is what makes the answer trustworthy. The bottling plant's quality manager acts on drift either way: under-filling short-changes customers, over-filling gives product away. So the test is two-tailed. A marketing team testing a new checkout may want to "prove it is better", but a worse checkout is exactly what they must catch: two-tailed again.

When this goes wrong

A one-tailed test is legitimate only when the opposite direction truly does not matter and that was decided in advance, in writing. A drug trial that can only license improvements can argue it; almost no business experiment can. And no choice of hypotheses rescues a vague question: "is the new page better?" must become a parameter (the conversion rate), a value (the old rate) and a direction before any test can answer it.

In code

scipy's alternative is chosen before you run the test, like the hypotheses. In Excel, B1 holds the t statistic from the data (chapter 6 computes it for the bottles): Excel has no one-sample t test function, so you compute t by formula and take its tail.