Multiple testing, pre-registration and p-hacking
Twenty metrics and one came up: did we find something? Forking paths, peeking and the corrections.
A tree grows from one experiment in which nothing works: both arms get the same page. It branches into 20 metrics, and each metric into 5 customer segments: 100 analyses, each a z test at α = 0.05. Every leaf whose p-value falls below 0.05 lights up as a "win". A line chart beside it climbs with the chance of at least one false win as the analyses add up: 5 % for one, 22.6 % for five, 40.1 % for ten, 64.2 % for twenty. At the end, Bonferroni, Holm and Benjamini–Hochberg are applied to the 100 p-values, and no win survives.
What you just saw
The animation ran an A/A test: both arms saw the same page, so nothing can have worked. Then it analysed the result the way an eager team might: 20 metrics, each split into 5 customer segments, 100 analyses in all. Four of them came out "significant" at α = 0.05. Each would make a convincing slide. Every one is a false win.
That is what a 5 % error rate means when it is applied many times. The line chart beside the tree shows the chance of at least one false win climbing with every analysis added: 22.6 % after five, 64.2 % after twenty, 99.4 % after a hundred. In 1,000 more A/A tests analysed the same way, 99.3 % produced at least one win. After a correction for testing many hypotheses at once, not one of the four survived.
How fast do false wins add up?
If each of m independent tests has a 5 % chance of a false alarm, the chance that at least one of them raises one, the family-wise error rate, is:
| Tests (m) | Chance of at least one false win at α = 0.05 |
|---|---|
| 1 | 5 % |
| 5 | 22.6 % |
| 10 | 40.1 % |
| 20 | 64.2 % |
| 100 | 99.4 % |
The expected number of false alarms is simply m × α. A team that runs forty tests a quarter at 5 % can expect about two false alarms every quarter, eight a year, even if nothing it tests ever works. Twenty metrics and one came up? That is exactly what nothing looks like.
Bonferroni, Holm and Benjamini–Hochberg
Ten metrics, four with a real effect: 10 p-values. At α = 0.05 without a correction 4 are rejected; Benjamini–Hochberg rejects 3, Holm 2 and Bonferroni 1.
A family's p-values, sorted from the smallest, are drawn as dots on a log scale, with four lines: the plain α of 0.05; Bonferroni's α divided by the number of tests, flat; Holm's steps, rising from Bonferroni's line towards α; and Benjamini–Hochberg's line, rising from α/m to α. A dot below a line is rejected by that method. For ten metrics of which four have a real effect, the plain α rejects four, Benjamini–Hochberg three, Holm two and Bonferroni one. A menu switches to Helion's six hypotheses, Meridian's six, and the 100 analyses of the A/A test.
Three corrections are in common use. Each sorts the family's p-values from the smallest and compares them with a threshold:
In words: \(p_{(i)}\) is the i-th smallest p-value and m the number of tests. Bonferroni divides α by m for every test. Holm does the same for the smallest p-value, then relaxes the threshold one step at a time, and stops at the first p-value that fails; it controls the same error rate as Bonferroni and never rejects fewer. Benjamini–Hochberg (BH) lets the threshold grow in proportion to the rank and rejects every p-value up to the largest one under its line.
Worked on the ten metrics, whose four smallest p-values are 0.0030, 0.0055, 0.0146 and 0.0259, at α = 0.05:
- Bonferroni's threshold is 0.05 ÷ 10 = 0.005: only 0.0030 is below it, so one rejection.
- Holm compares 0.0030 with 0.05 ÷ 10 = 0.005 (reject), 0.0055 with 0.05 ÷ 9 = 0.00556 (reject), 0.0146 with 0.05 ÷ 8 = 0.00625 (fail, and stop): two rejections.
- BH compares the i-th smallest with i × 0.005: 0.0146 is under 3 × 0.005 = 0.015, but 0.0259 is over 4 × 0.005 = 0.020, and nothing further down passes either: three rejections.
They control different things. Bonferroni and Holm control the family-wise error rate: the chance of any false positive. BH controls the false discovery rate: the expected share of the rejections that are false. Analysis plans often call BH's α "family-wise", even when they pre-register it carefully; that is not what it controls. BH accepts a few false leads in exchange for missing fewer real effects, which suits exploratory work; a regulator's confirmatory test needs Holm or Bonferroni.
| Method | Controls | Behaviour | When to use |
|---|---|---|---|
| Bonferroni | The family-wise error rate: the chance of any false positive | Divide α by the number of tests. Very conservative; costs real power | Confirmatory work; regulatory settings; few tests |
| Holm | The family-wise error rate | Step down from α/m towards α; never less powerful than Bonferroni | Whenever you would use Bonferroni |
| Benjamini–Hochberg | The false discovery rate: the expected share of rejections that are false | Rank the p-values and apply a graduated threshold. Less conservative | Exploratory analytics, where a few false leads are tolerable but missed effects are not |
For Helion's six hypotheses the choice does not matter: BH, Bonferroni and Holm all reject H1 to H5 and keep H6. For Meridian's six all three reject every hypothesis except H2. That is the comfortable case, and the lab computes it rather than typing it: when they disagree, which typically happens for a result between 0.01 and 0.05, say so explicitly rather than quoting whichever supports the recommendation.
Peeking: the p-value is not a progress bar
First, one A/A test (no real difference) is checked ten times as its data come in: its p-value wanders from look to look, dips below 0.05 at one look, and ends well above it. An analyst who stops at the first p below 0.05 declares a win; one who waits for the planned end finds nothing. Then 20,000 A/A tests are run the same way and two counters fill: the share that crossed 0.05 at any of the ten looks, about 19 %, and the share significant at the planned final look, about 5 %.
Peeking is multiple testing in time. The p-value of a test with no real effect wanders as the data come in, and at any one look it has a 5 % chance of dipping below 0.05. Look ten times and stop at the first dip, and the chance of a false win is not 5 %: in the animation's 20,000 A/A tests it was 19.1 %. (Ten looks are often said to push the rate "above 20 %"; about 19 % is right, and it keeps climbing with more looks.) Waiting for the planned final look kept it at 5.1 %, as promised.
The fix is to fix the sample size in advance and look once, or to use a sequential design built to be looked at (group-sequential boundaries or always-valid p-values, which spend α across the looks). Those designs are beyond this lab; the rule is not.
The garden of forking paths
The bottles of chapter 6 did not reject: 25 bottles, a two-tailed test at α = 0.05, p = 0.057. Now change one choice at a time:
| Change | New result | Why |
|---|---|---|
| n = 50 instead of 25 (same x̄ and s) | t = 2.83, p = 0.0068: reject H₀ | The standard error falls to 1.41; the same 4 ml gap is now nearly three standard errors out |
| A one-tailed test, Hₐ: μ > 500 | p = 0.0285: reject H₀ | All 5 % sits in one tail. Legitimate only if the direction was chosen in advance |
| α = 0.10 instead of 0.05 | p = 0.057 < 0.10: reject H₀ | A looser threshold, accepting more false positives |
This table is uncomfortable, and it should be. The same 25 bottles support opposite conclusions depending on decisions about sample size, tail and α. That is precisely why those decisions must be made and written down before the data are collected. Choosing them afterwards, to land on the answer you wanted, is p-hacking, and it is usually committed in good faith by someone who does not realise that is what they are doing. Every choice (which metric, which segment, which outliers to drop, when to stop) is a fork in the path, and an analyst who walks the garden of forking paths looking for significance will find it.
Pre-registration: the cheapest guardrail
The fix is procedural, not statistical. Pre-registration commits the analysis plan, as structured data rather than prose, before the experiment starts. Helion's plan for Portal v2 was committed on 2 June 2026, before enrolment opened:
| Field | Value | What it fixes |
|---|---|---|
experiment_id | EXP-2026-014-PORTALV2 | Which experiment this plan belongs to |
primary_metric | went_paperless | The one metric the decision rests on |
alpha | 0.05 | The significance level, chosen before the data |
power_target | 0.80 | The power the sample was sized for |
mde_paperless_pp | 3.0 | The smallest lift worth funding, in percentage points |
multiplicity_method | fdr_bh | Benjamini–Hochberg across the family: it controls the false discovery rate |
H1 and H2 tail | two-sided | Both primary hypotheses can report harm as well as benefit |
A commit timestamp is proof that the hypotheses were not chosen after seeing the results. It costs nothing, and it is the difference between an analysis and a rationalisation.
Six rules that matter more than the code
- No peeking. Do not run the readout daily and stop when p first drops below 0.05. Repeated looks inflate the false-positive rate: ten looks turn a nominal 5 % into about 19 %. Fix the sample size in advance, or adopt a sequential design built to be peeked at.
- One primary metric. Secondary metrics are for diagnosis and new hypotheses. Promoting one to primary after the real primary disappoints is the most common form of results-shopping in industry.
- Effect size is mandatory. No result reaches a business audience without an interval and an explicit comparison with the decision threshold.
- Segment analyses are exploratory unless pre-registered. Slicing a null result by region, then segment, then tenure until something turns significant will always succeed in the end: that is what a 5 % error rate applied twenty times means. Findings from unplanned slicing are hypotheses for the next experiment, never conclusions from this one.
- Publish null results. A file drawer full of failed experiments distorts an organisation's picture of what works, exactly as publication bias distorts a scientific literature.
- Separate statistical significance from the business decision. The test says whether chance is a plausible explanation. Whether to spend €4m is a judgement about the interval, the cost, the risk and the strategy. Analysts who conflate the two get overruled; analysts who present them separately get trusted.
Why it matters
Every dashboard with twenty metrics, every experiment sliced by segment and every test checked daily is a multiple-testing problem. Without a plan and a correction, an organisation will ship changes that do nothing, one convincing slide at a time, and wonder a year later why the benefits never appeared.
When this goes wrong
Corrections cost power: with Bonferroni across 100 tests, a real effect needs p below 0.0005 to count, so plan fewer, better hypotheses rather than correcting many. The formula for the family-wise rate assumes independent tests; correlated metrics (revenue and orders, say) inflate it less, which makes Bonferroni conservative. And no correction rescues a garden of forking paths walked after the fact: you cannot correct for the analyses you tried and did not report.
In code
| What it computes | Python (scipy, statsmodels) | Excel |
|---|---|---|
| The three corrections on one family of p-values | multipletests(p_values, alpha=0.05, method='fdr_bh') # or 'bonferroni', 'holm' |
=IF(B2<=RANK.EQ(B2, $B$2:$B$11, 1)*0.05/10, "pass", "fail") |
| The chance of at least one false win in m tests | 1 - (1 - 0.05) ** 20 |
=1-(1-0.05)^20 |
| Bonferroni's threshold for m tests | 0.05 / m |
=0.05/10 |
multipletests is in statsmodels.stats.multitest; it returns the decisions and the adjusted p-values. The Excel line is the Benjamini–Hochberg comparison for each p-value; BH then rejects every p-value up to the largest that passes, even one that failed on its own.
Quiz
Three questions on the chapter. Your best score is kept in this browser.