Tool

Multiple testing simulator: forking paths and peeking

Set-up

20 metrics

5 segments

Significance level

Seed

One A/A test, no real effect anywhere, analysed once per metric and segment.

Playback speed

4 false wins in 100 analyses. In 1,000 more A/A tests like it, 99.3 % had at least one (for independent tests, 1 − (1 − α)^m = 99.4 %). After correction: Bonferroni 0, Holm 0, Benjamini–Hochberg 0.

This run's p-values against the corrections

Sorted p-values against the lines of the three corrections This A/A test: 100 p-values. At α = 0.05 without a correction 4 are rejected; Benjamini–Hochberg rejects 0, Holm 0 and Bonferroni 0. This A/A test: p-values from the smallest, log scale 1 10⁻² 10⁻⁴ 10⁻⁶ 10⁻⁸ 10⁻¹⁰ 10⁻¹² α = 0.05 Benjamini–Hochberg Holm Bonferroni rejected by BH ▼ marks a p-value smaller than 10⁻¹², drawn at the floor.

This A/A test: 100 p-values. At α = 0.05 without a correction 4 are rejected; Benjamini–Hochberg rejects 0, Holm 0 and Bonferroni 0.

Every analysis choice is another chance to find something that is not there. Run an A/A test (no real effect anywhere) sliced into many metrics and segments, or peek at one test every day and stop at the first p below 0.05, and count the false wins. Then apply Bonferroni, Holm and Benjamini–Hochberg to the same p-values.

The same in code

Python (statsmodels)
multipletests(p_values, alpha=0.05, method='fdr_bh')  # 'bonferroni', 'holm'
Chance of at least one false win
1 - (1 - 0.05) ** 100
Excel
=1-(1-0.05)^100
An unhandled error has occurred. Reload 🗙

Reconnecting…

Your connection to Hypothesis Testing Lab dropped, probably a Wi-Fi blip. Hold on while it reconnects.

The connection could not be restored. The page will reload in a few seconds.