One proportion

What you just saw

When the outcome is yes or no (on time or late, converted or not, a large loss or not) you are testing a proportion. A delivery firm promises that 95 % of deliveries arrive on time; this month 470 of 500 did, 94 %. Is it slipping?

The bars are the null world, exactly: if the promise holds, the number on time in 500 deliveries follows a binomial distribution, centred on 475. The shaded bars are the months at least as far from 475 as 470 is, in either direction: their total is the exact p-value, 0.304. The curve is the normal approximation to those bars, and its shaded tails give almost the same answer, 0.305. With 475 expected on time and 25 expected late, the curve fits the bars well.

Now choose the small audit: a supplier promises that at most 2.5 % of parts are faulty, and an audit finds 4 faulty in 60. Only 1.5 faulty parts are expected, and the bars are a lopsided staircase the symmetric curve cannot follow. The z test says p = 0.039 and would reject; the exact test says p = 0.063 and does not. When the check fails, trust the bars.

The one-proportion z test

The test statistic is the familiar signal over noise, with the noise worked out from the promised proportion:

In words: \(\hat{p}\) is the observed proportion (470 / 500 = 0.94), \(p_0\) the proportion H₀ claims (0.95) and \(n\) the number of trials (500). The bottom is the standard error of a proportion if H₀ is true: a proportion's spread is fixed by the proportion itself, so no separate spread is estimated. For the deliveries:

The observed rate is about one standard error below the promise: p = 0.305, so we fail to reject H₀. A one-point shortfall in 500 deliveries is well within what chance produces. Use it when you compare an observed rate with a fixed target: is our on-time rate still the 95 % we promise?

The standard error uses \(p_0\), not \(\hat{p}\), because the test works inside the null world, where the proportion is \(p_0\). The usual interval for a proportion uses \(\hat{p}\) instead, which is why chapter 5 found that a test and an interval for a proportion can disagree near the boundary.

The exact binomial test

The z test borrows the normal curve. The exact binomial test does not: it adds up the binomial probabilities of every outcome at least as extreme as the one observed, exactly as the null world of chapter 1 did. The rule of thumb for when the normal curve can stand in for the bars is at least 10 expected successes and 10 expected failures:

For the deliveries, 475 and 25: safe. For the audit, 1.5 and 58.5: not. The courier of chapter 1 is close to the edge, with 16 late parcels expected: its one-sided exact p-value is 0.0306, the z test's 0.0185. Both are below 0.05, but the z test's p-value is about 40 % smaller: it overstates the evidence. With a computer at hand there is little reason not to use the exact test.

Meridian's large losses: a tariff under test

Meridian Assurance, a UK motor insurer, prices every premium with a large-loss loading built on one assumption: 2.5 % of claims exceed £25,000. That single number feeds every premium the company charges, so if experience says otherwise, the tariff is mispriced. The threshold and the rate come from the filed tariff, not from this analysis: an auditor will want to see that they were not chosen after the fact.

In accident year 2025, 171 of 4,663 settled claims exceeded £25,000: 3.67 %, against the tariff's 2.5 %. Under H₀ we would expect 116.6. The exact binomial test gives p = 1.6 × 10⁻⁶: the tariff's assumption is contradicted. How far off is it? The exact interval says the true large-loss rate is between 3.15 % and 4.25 % with 95 % confidence.

That interval is a Clopper–Pearson interval: the range of proportions for which the exact test would not reject. It is built from the bars, not from a curve, so it is guaranteed to cover the true rate at least 95 % of the time. With money attached to the answer, the exact method is the defensible one. Here the normal approximation would have been safe (116.6 expected), but it is poorest in exactly the far tail where this result sits: it puts the p-value at 3.3 × 10⁻⁷, five times smaller.

Why it matters

Promises and tariffs are proportions: 95 % on time, 2.5 % large losses, 1 % defects. A test of one proportion says whether this month's shortfall is a signal or the noise any month has, before anyone rewrites a contract or a price. Using the exact test when counts are small keeps a supplier from being penalised on four parts.

When this goes wrong

Every version of this test assumes independent trials: one delivery's fate says nothing about the next. Deliveries on the same van on the same snowy day are not independent, and neither are claims from the same storm; the test then understates the noise. The claim data have a subtler problem too: large bodily-injury claims take years to settle, so a count of settled claims within the year understates large losses, and in the direction that makes the finding worse, not better.

In code

proportions_ztest is in statsmodels.stats.proportion (pass prop_var to use the null proportion in the standard error, as the formula does); binomtest is in scipy.stats. Excel's BINOM.DIST gives one tail; the exact two-sided p-value adds every outcome no more likely than the observed one, which Excel has no single function for. BETA.INV gives the Clopper–Pearson lower bound; the upper is =BETA.INV(0.975, 172, 4663-171).