The null world

What you just saw

A courier promises that 92 % of parcels arrive on time. This month, 176 of 200 did: 88 %. Is the courier slipping, or was it just a bad month?

The animation took the promise at its word. It played 1,000 months in a world where every parcel is on time with probability 0.92, and piled up how many of the 200 arrived on time in each. That pile is the null world: what months look like when nothing has changed. Most months land within about 8 parcels of the 184 the promise predicts.

Then the real month landed at 176, and every simulated month at least as bad lit up: 31 of the 1,000, a share of 0.031. In words: if the courier really kept its promise, a month this bad or worse would turn up about 3 times in 100. That share has a name, the p-value, and chapter 3 is about it. You have just worked one out without a formula.

What hypothesis testing is for

Every interesting claim about the world is made about a population: all your customers, all the bottles coming off a line, all the parcels a courier delivers. You almost never measure the population. You measure a sample and try to reason back to the population.

That backwards step is the whole problem. A sample differs from its population by chance alone. If you weigh 25 bottles and the average comes out at 504 ml when the machine is really set to 500 ml, nothing has gone wrong: samples wobble. So when you see a difference, you face a question with real consequences:

Hypothesis testing is an agreed procedure for answering that. It does not tell you what is true. It tells you whether your evidence is strong enough to stop believing that nothing is going on.

Building the null world

To ask whether luck could explain a result, you need to know what luck alone produces. So you suppose the dull explanation is right, here that the courier's rate really is 92 %, and work out what the data would look like. That imagined world is the null world, and the dull explanation is the null hypothesis.

Here the null world has a familiar shape. Each parcel is on time or not, with the same chance and independently of the others, so the number on time out of 200 follows the binomial distribution. (For a refresher, the Probability Lab's chapter on the binomial covers it.)

Two numbers describe it. The count we expect is \(np = 200 \times 0.92 = 184\), and the typical distance from it, the standard deviation, is \(\sqrt{np(1-p)} = \sqrt{200 \times 0.92 \times 0.08} = 3.84\) parcels. So 176 sits 8 below what the promise predicts: a little over two standard deviations.

The question is how often the null world produces a month at least that bad. Adding up the binomial probabilities of 176, 175, 174 and every count below answers it exactly:

Read it symbol by symbol. \(X\) is the number on time in a month where the promise holds; \(\binom{200}{k}\) counts the ways of choosing which k of the 200 parcels were on time; \(0.92^{k}\, 0.08^{200-k}\) is the chance of any one of those ways. The sum is 0.0306: about 3 months in 100.

Reading the share

The simulation gave 0.031 and the exact sum 0.0306. They differ because 1,000 simulated months are a sample too: with more months the share settles on the exact value (the law of large numbers). Either way the message is the same. Luck alone produces a month this bad only about 3 times in 100.

Be precise about what the share is. It is the chance of a month at least this bad if the promise holds. It is not the chance that the promise holds. The first is worked out inside the null world; the second would need to know how often couriers break their promises, which this calculation never used. Chapter 3 comes back to this, the commonest misreading in statistics.

Why it matters

The contract has a penalty clause for a courier that misses its promise. Invoking it for a month that luck alone often produces would be unfair and would sour the relationship; shrugging off a month that luck produces only 3 times in 100 may let a real slip run on. The null world turns "that looks bad" into a number you can weigh against the cost of each mistake. Chapter 2 writes the question down properly, chapter 3 sets the bar before looking, and chapter 4 counts how often each kind of mistake happens.

When this goes wrong

The null world is only as good as its model. The binomial assumes that parcels are late independently of one another. A snowstorm that holds up a whole van breaks that: late parcels come in clusters, real months vary more than the binomial says, and the share of null months at least as bad comes out too small. A result can look surprising only because the null world was drawn too narrow.

A small share also says only that luck is an unlikely explanation, not what the explanation is. The courier may be slipping, or this month may simply have been harder: more rural addresses, a bank holiday, a strike. The test tells you that something beyond chance is probably going on; finding out what is the analyst's job.

In code

In Excel, fill =BINOM.INV(200, 0.92, RAND()) down 1,000 rows (A2:A1001) to build the null world; it redraws on every recalculation. binom and binomtest are in scipy.stats, and np is numpy. The simulated share differs a little from run to run; the exact one does not.