Two proportions

What you just saw

Helion's portal test had a second question, the one the business case rests on: does Portal v2 get more customers to switch to paperless billing? Of 6,031 customers on Portal v2, 2,290 went paperless (37.97 %); of 5,969 on the old portal, 2,052 did (34.38 %). A gap of 3.59 percentage points.

The top panel shows the two rates with their intervals. The lower panel asks the test's question: in a world where the two portals convert equally, how far apart would two rates like these land by chance? The curve is that null world, and the observed gap sits far out in its tail: the shaded area is 4.2 × 10⁻⁵. The bar under it is the answer to the question a decision needs: the true gap is plausibly anywhere from 1.87 to 5.31 points.

Switch to the small pilot and the picture changes: 138 against 110 of 1,000, a gap of 2.8 points, and the curve is so much wider that the gap is no longer surprising (p = 0.057). Switch to the tiny trial and the z test's curve can no longer be trusted at all.

The two-proportion z test

The two-proportion z test is the workhorse of A/B testing. If H₀ is true the two groups share one proportion, so the test estimates it from both groups together, the pooled proportion, and uses it for the noise:

In words: \(x_1\) and \(x_2\) are the successes in the two groups, \(n_1\) and \(n_2\) their sizes, \(\hat{p}_1\) and \(\hat{p}_2\) their rates and \(\hat{p}\) the pooled rate. For Helion the pooled rate is 4,342 paperless customers in 12,000, 0.3618, and the standard error of the gap is 0.00877:

The gap is about four standard errors from zero, p = 4.2 × 10⁻⁵: we reject H₀. Portal v2 does move paperless adoption. It assumes independent observations and enough data for the normal approximation: the usual rule is at least 10 successes and 10 failures in each group.

Two standard errors, one for each job

The test and the interval answer different questions, so they use different standard errors. The test works inside the null world, where the two rates are equal, so it uses the pooled rate. The interval describes the gap as it is, so it uses each group's own rate:

For Helion the two hardly differ (0.00877 both, to three significant figures), so the test and the interval agree comfortably. They need not. When the rates are far apart or the groups very different in size, the two standard errors part company, and near the boundary the test can reject while the interval still contains zero, or the other way round (chapter 5 showed it for one proportion). statsmodels' default interval, Newcombe's, is built differently again; for Helion it also runs from 1.87 to 5.31 points.

Read the interval, not the point

The p-value is emphatically below 0.05, so we reject H₀. But the decision does not turn on that. "Portal v2 converts 3.59 more customers per 100" is only the point estimate: the data support anything from about 1.87 to 5.31 more per 100. Finance will not fund the rollout for a lift below 3 points, and the interval straddles that threshold. Chapter 16 takes that reading to the board.

How big is it? Cohen's h

A gap of 3.59 points means more between 34 % and 38 % than between 1 % and 5 %, so proportions have their own standardised effect size, Cohen's h, the gap measured on a scale that stretches the ends:

By the usual benchmarks (0.2 small, 0.5 medium, 0.8 large) an h of 0.075 is tiny. That is not a contradiction of the tiny p-value: with 12,000 customers the test can detect effects far smaller than anything worth acting on, so significance has stopped being informative and the interval has taken over the job.

The small pilot: a real-looking gap this sample cannot confirm

Suppose Helion had first run a pilot with 1,000 customers per portal: 138 against 110 went paperless, 13.8 % against 11.0 %. The gap of 2.8 points looks like a win. The test says z = 1.90, p = 0.057: not significant at 5 %. The 95 % interval, −0.09 to 5.69 points, holds both zero and a lift worth funding. This pilot cannot tell them apart; it can only say "not shown", never "no effect". Chapters 16 and 17 are about exactly this: how to read such a result, and how many customers a test needs before it starts.

Small counts: Fisher's exact test

When a group has fewer than 10 successes or failures, the normal curve under the z test is no longer trustworthy, as it was not for one proportion in chapter 9. Fisher's exact test works with the counts directly: given the table's totals, it adds the exact probabilities of every table at least as extreme as the observed one. Take a tiny trial of two call scripts, 15 calls each: 6 sales with the new script, 1 with the old.

The z test says z = 2.16, p = 0.031, and would declare the new script better. Fisher's exact test says p = 0.080: not significant. With one sale in one group, the exact answer is the honest one. For large samples the two agree: on Helion's 12,000 customers Fisher's p is 4.4 × 10⁻⁵, against the z test's 4.2 × 10⁻⁵.

Why it matters

Conversion, adoption, churn, default: most A/B tests are two proportions. The test says whether a gap is real; the interval says whether it is big enough to pay for the change, and here that is the question that decides the rollout. At €8.40 a year for each customer who goes paperless, the difference between 1.87 and 5.31 points across 4.2 million customers is the difference between a marginal and a strong business case.

When this goes wrong

The z test assumes every customer is an independent trial. Customers randomised in households, or counted twice, break that and make the standard error too small. With few successes in a group the normal approximation fails, as the tiny trial showed: use Fisher's exact test. And a test that stops early, the first day the p-value dips below 0.05, is not a 5 % test any more (chapter 18).

In code

proportions_ztest, confint_proportions_2indep and proportion_effectsize are in statsmodels.stats.proportion; fisher_exact and chi2_contingency in scipy.stats. Excel has no two-proportion test: the formulas above build it. HYPGEOM.DIST gives the probability of one table; Fisher's p-value adds those of every table at least as extreme. CHISQ.TEST needs the expected table in E2:F3, built from the margins, and applies no correction.