Run one properly and tell the board: pre-registration, gates, six hypotheses and the decision memo.
The experiment pipeline: twelve steps from pre-registration to the memo
All 12 layers built: 1. Pre-register (tails, α and BH, in git); 2. Quality gates (grain, gaps, plausibility); 3. SRM check (χ² on the 50/50 split); 4. H1 paperless (two-proportion z); 5. H2 days to payment (Welch's t); 6. H3 calls per agent (paired t); 7. H4 regions (ANOVA and Tukey); 8. H5 segment × payment (χ² independence); 9. H6 channel mix (χ² goodness of fit); 10. The family (BH, Bonferroni, Holm); 11. Power (n and MDE for next time); 12. The memo (the decision for the board).
A pipeline diagram in three rows. Before the data: pre-register, the quality gates, the sample ratio mismatch check. The hypotheses: H1 paperless adoption (two-proportion z), H2 days to payment (Welch's t), H3 calls per agent (paired t), H4 regions (ANOVA and Tukey), H5 segment by payment method (χ² independence) and H6 channel mix (χ² goodness of fit). After the tests: the family of six p-values corrected together, power for the next experiment, and the memo. Buttons build it up one layer at a time; each layer is a step of this chapter.
What you just saw
The picture is the whole of this chapter: one experiment readout as a pipeline of twelve steps. Three run before any hypothesis is tested, six each test one hypothesis with a different test, and three turn the results into a decision. Build it up a layer at a time, or follow the chapter, where each step adds its layer.
Helion Energy is a fictional European energy retailer with 4.2 million customers. It has rebuilt its self-service billing portal. Before committing to a full rollout, leadership randomised 12,000 customers into control (the old portal) and treatment (Portal v2) for one billing cycle.
The business case rests on two claims: that Portal v2 increases paperless billing adoption, saving roughly €8.40 per converted customer per year, and that it reduces days to payment, which improves working capital. Finance has said it will not fund the rollout for a paperless lift below 3 percentage points.
Six questions must be answered, and this is the point of the example: each one is a different shape of data and therefore a different test.
Six questions, six tests: the mapping is the skill, the arithmetic is the library's job
#
Business question
Outcome type
Test
H1
Does Portal v2 lift paperless adoption?
Yes or no, two independent groups
Two-proportion z test
H2
Does Portal v2 reduce days to payment?
A number, two independent groups
Welch's two-sample t test
H3
Did calls per agent fall after the rollout?
A number, paired
Paired t test
H4
Does days to payment differ by region?
A number, four groups
One-way ANOVA and Tukey's HSD
H5
Is segment associated with payment method?
Two categories
χ² test of independence
H6
Does the channel mix match the capacity forecast?
One category against expected shares
χ² goodness of fit
The datasets
The data are the notebook's own, exported row for row. Download them to follow each step in Python or Excel; the README says what every column holds.
helion-portal-v2-customers.csv· 12,000 rowsOne row per enrolled customer, randomised between the old portal (control) and Portal v2 (treatment) for one billing cycle.
helion-agent-calls.csv· 80 rowsOne row per call-centre agent, measured in the four weeks before the rollout and the four weeks after.
helion-channel-mix.csv· 4 rowsOne row per contact channel over the experiment window, with the share the capacity plan forecast.
Every number in this chapter is computed by the lab's engine from these files and checked against scipy and statsmodels. Where Charles's notebook or tutorial said something the numbers do not support, a correction box shows what it said and what is right.
One experiment, start to finish
One A/B test, start to finish: the simulator
The assignment service sends 45 % to the old portal and 55 % to Portal v2: the SRM gate halts the readout.
Playback speed
H1 in the family, after the gate
Hypothesis
p
Benjamini–Hochberg adjusted
Bonferroni adjusted
No readout: the sample ratio mismatch gate halted it before H1 was tested.
A chart fills in day by day over a 30-day billing cycle as customers enrol and split between the two portals. It draws the running lift in paperless adoption, in percentage points, with its 95 % interval as a band that narrows as customers arrive, against a line at zero and a dashed line at finance's 3-point threshold. When enrolment closes, the sample ratio mismatch gate checks the split at α = 0.001; if it fails, the readout halts. Otherwise H1 is tested and the decision, the interval and the smallest lift this size could detect are written out. Presets set up the tutorial's five exercises; each run is seeded, so it replays the same.
The simulator plays one experiment through the pipeline, drawn afresh from the notebook's own generator: customers enrol day by day and split between the two portals; the running lift and its interval narrow as they arrive; when enrolment closes the sample ratio mismatch gate checks the split, and only then is H1 tested. A fresh draw lands somewhere else than the notebook's own run, which is the point of the second preset. The section before the memo uses the presets for the tutorial's five exercises.
A textbook stops at the p-value. A production readout has to survive an audit, a finance challenge, and a post-mortem eighteen months later when someone asks why the benefit never materialised. Seven additions do that work:
Pre-registration: hypotheses, tails, α and the primary metric are committed to version control before the data are opened.
Data quality gates: grain, completeness and plausibility checks that stop the pipeline rather than print a warning.
A sample ratio mismatch test: checking that the randomisation itself worked before trusting anything after it.
Documented assumption checks, with the fallback test named in advance, so the choice is not made after seeing which one gives the nicer answer.
Effect sizes and confidence intervals on every result, because "significant" and "worth doing" are different claims.
Multiplicity control: six tests at 5 % is not a 5 % error rate.
Power analysis, so the next experiment is sized deliberately rather than by whatever traffic happened to show up.
Step 1: pre-register before you look
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: step 1, pre-registration, is marked as the current one, and the rest are still to come.
Chapter 6 ends with an uncomfortable table: the same 25 bottles support opposite conclusions depending on choices about the tail, α and the sample size. In a business that flexibility is dangerous, because there is always a stakeholder who wants a particular answer, and always an analyst who can find a defensible route to it without ever consciously deciding to.
The fix is procedural, not statistical. Commit the analysis plan to version control before the experiment starts, as structured data rather than prose: a pre-registration. A commit timestamp is proof that the hypotheses were not chosen after seeing the results. It costs nothing, and it is the difference between an analysis and a rationalisation.
The pre-registration as data, committed before enrolment opened
Why both primary hypotheses are two-sidedA one-sided test would have more power to detect the improvement everyone is hoping for. It would also be structurally incapable of reporting harm, and a redesigned portal that makes customers pay later is precisely the outcome the experiment exists to catch. For a go/no-go decision on a change that could plausibly move either way, two-sided is the only honest choice.
Fixing a two-sided test at α = 0.05 also fixes the cut-off before any data arrive: H1 will be rejected if |z| is larger than 1.96, whatever anyone hopes.
Pre-register the tail of H1
Finance hopes Portal v2 raises paperless adoption. Before the data are opened, which alternative should the pre-registration give H1?
What it computes
Python (scipy, statsmodels)
Excel
The two-sided cut-off the pre-registration fixes
norm.ppf(1 - 0.05 / 2)
=NORM.S.INV(1-0.05/2)
The plan read back, so the analysis uses what was committed
In the Excel line, B1 holds α as committed in the plan.
Step 2: the quality gates, before any statistics
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: step 1 is built, step 2, the quality gates, is marked as the current one, and the rest are still to come.
Three quality gates run before a single hypothesis is tested. In the scheduled pipeline they raise an error and stop the run rather than print a warning, because testing hypotheses on broken data produces a confident wrong answer, which is considerably worse than no answer.
The three gates on Helion's data: all pass
Gate
What it checks
On Helion's data
1. Grain
One row per customer
0 duplicated customer ids in 12,000 rows
2. Completeness
No empty variant, went_paperless or days_to_payment
0 empty cells
3. Plausibility
No negative days to payment
0 negative values
Gate 1 deserves particular attention, because it enforces the one assumption no statistical test can detect. Every test in this chapter assumes independent observations. If a customer appears twice (a duplicated join, a re-enrolment, a fan-out from a one-to-many relationship), the effective sample size is smaller than the row count, every standard error is too small, and every p-value is too small. No normality test will hint at this. The grain check is the only thing standing between you and a false discovery.
The Excel lines assume the customers file opened with its columns in order, A to G.
Step 3: sample ratio mismatch, the first test to run
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 2 are built, step 3, the sample ratio mismatch check, is marked as the current one, and the rest are still to come.
The split: customers on each portal against a fair 50/50
Two pairs of bars: the old portal had 5,969 customers and Portal v2 6,031, each beside the 6,000 a fair 50/50 split expects. The gaps are small: χ² = 0.320, p = 0.571.
Before asking whether the treatment worked, ask whether it was assigned fairly. The experiment randomised 50/50, so the observed split should be consistent with 50/50. That is exactly a χ² goodness-of-fit test with two categories, and a failure of it is a sample ratio mismatch. H₀: the assignment ratio is 50/50. Hₐ: it is not.
Here \(O\) is each arm's observed count and \(E\) the count a fair split expects, 6,000 each. So \(\chi^2 = (5{,}969 - 6{,}000)^2/6{,}000 + (6{,}031 - 6{,}000)^2/6{,}000 = 0.160 + 0.160 = 0.320\) on 1 degree of freedom, and p = 0.571. No mismatch: safe to proceed.
Two things are inverted here: notice bothWe want a large p-value. A small one means the randomiser, the event logging or an upstream filter is broken, and every result after it would measure selection bias rather than the treatment. And the check uses a stricter α, 0.001, not the usual 0.05.
The SRM check's power at 12,000 customers: a stricter α catches fewer mismatches (C28)
True split
Caught at α = 0.001
Caught at α = 0.05
51/49
14 %
59 %
52/48
86 %
99 %
The SRM statistic by hand
Helion randomised 50/50 and got 5,969 customers on the old portal and 6,031 on Portal v2. What is the χ² goodness-of-fit statistic against 6,000 each? Three decimals.
A decimal or a fraction; within 0.001 of the answer counts as right.
In Excel, B2:C2 hold the observed counts and B3:C3 the expected 6,000 each. The power formula is exact here: with two categories, χ² is the square of a z whose shift is w√n, and w = 0.04 for a 52/48 split. GofChisquarePower is in statsmodels.stats.power.
Step 4: H1, paperless adoption, and the trap in a large sample
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 3 are built, step 4, H1, paperless adoption, is marked as the current one, and the rest are still to come.
H1: the lift in paperless adoption, its interval and finance's threshold
One interval on a scale of percentage points: the lift of 3.59 points with its 95 % interval from 1.87 to 5.31. A solid line at 0 marks no difference, and a dashed line at 3 marks finance's funding threshold. The interval is far from 0, so the lift is real, but it straddles 3, so it may or may not be large enough to fund.
A two-proportion z test on 12,000 customers:
H1's results: read the interval, not the p-value
Quantity
Value
Old portal adoption
34.38 % (n = 5,969)
Portal v2 adoption
37.97 % (n = 6,031)
Absolute lift
3.59 percentage points
z statistic
4.095
p-value
4.2 × 10⁻⁵
95 % interval on the lift
1.87 to 5.31 percentage points
Cohen's h
0.075 (very small)
Under H₀ both portals share one adoption rate, so the test pools the two arms to estimate it and uses that for the standard error:
Symbol by symbol: \(x_1 = 2{,}290\) of the \(n_1 = 6{,}031\) Portal v2 customers went paperless, and \(x_2 = 2{,}052\) of the \(n_2 = 5{,}969\) on the old portal. The pooled rate is \(\hat{p} = 4{,}342 / 12{,}000 = 0.3618\), the standard error is \(\sqrt{0.3618 \times 0.6382} \times \sqrt{1/6{,}031 + 1/5{,}969} = 0.00877\), and \(z = 0.03593 / 0.00877 = 4.095\): the gap is about four standard errors from zero. The interval uses each arm's own rate for its standard error instead, because it is not computed under H₀ (chapter 5 says when that matters).
Why z and not t? A proportion's variance is a function of the proportion itself, so there is no separate spread to estimate and nothing to fatten the tails. With at least 10 successes and 10 failures in each group, the normal distribution applies directly.
The p-value is emphatically below 0.05, so we reject H₀: Portal v2 does move paperless adoption. But the decision does not turn on that, and this is where most enterprise A/B readouts go wrong.
Cohen's h of 0.075 is tiny by conventional benchmarks. That is not a contradiction: with 12,000 customers the test has enough power to detect effects far smaller than anything worth acting on, so statistical significance has stopped being informative and the confidence interval has taken over the job.
The finding that actually goes to the boardThe true lift is somewhere between 1.9 and 5.3 percentage points. Finance set the funding threshold at 3. The interval straddles that threshold, so the honest conclusion is "real, and plausibly but not certainly large enough to fund": a materially different message from "significant, ship it".
Translated into money across 4.2 million customers at €8.40 each, the annual benefit is somewhere between €0.66m and €1.87m, with a point estimate of €1.27m. The business case should be underwritten at the lower bound, not the point estimate.
The benefit at the interval's lower end
H1's 95 % interval for the lift runs from 1.87 to 5.31 percentage points. With 4.2 million customers and €8.40 saved per converted customer per year, what is the annual benefit at the interval's lower end, in € millions? Two decimals.
A decimal or a fraction; within 0.01 of the answer counts as right.
proportions_ztest, confint_proportions_2indep and proportion_effectsize are in statsmodels.stats.proportion. In Excel, B1 holds the z from the line above. The interval's upper end replaces the minus sign after the lift with a plus.
Step 5: H2, days to payment, with the assumptions driving the test
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 4 are built, step 5, H2, days to payment, is marked as the current one, and the rest are still to come.
H2: the difference in days to payment, beside the smallest difference the design could detect
On a scale of days, Welch's estimate of the difference, Portal v2 minus the old portal, is −1.09 days with a 95 % interval from −1.21 to −0.97. Below it, the Mann–Whitney check's shift in medians, −1.06 days, shown as a point. A line marks zero, and a shaded band from −0.18 to 0.18 days marks differences too small for this design to detect reliably. The whole interval sits far outside it.
A numerical outcome in two independent groups. The notebook checks the assumptions first and lets the checks decide how far to trust the test, rather than picking a test and hoping.
Assumption checks for H2
Check
Result
Consequence
Levene's equal-variance test
W = 1.23, p = 0.267
Variances comparable
Skewness of the outcome
+0.795: right-skewed, as billing data usually are
The raw data are not normal, but at about 6,000 per group the sampling distribution of the mean is close to normal, so this does not disqualify the test
Group sizes
5,969 and 6,031
Ample
Welch's test is used whatever Levene says: it is valid whether or not the variances are equal, and it costs almost nothing when they are (chapter 7's "just use Welch").
Portal v2's customers paid in \(\bar{x}_1 = 14.36\) days on average (\(s_1 = 3.40\), \(n_1 = 6{,}031\)) and the old portal's in \(\bar{x}_2 = 15.44\) (\(s_2 = 3.48\), \(n_2 = 5{,}969\)). The standard error of the gap is \(\sqrt{3.40^2/6{,}031 + 3.48^2/5{,}969} = 0.0628\) days, so \(t = -1.09 / 0.0628 = -17.35\) on about 11,986 degrees of freedom (Welch's formula for them is in chapter 7).
H2's results
Quantity
Value
Old portal mean
15.44 days
Portal v2 mean
14.36 days
Difference
−1.09 days
t statistic (Welch)
−17.35, df ≈ 11,986
p-value
1.3 × 10⁻⁶⁶
95 % interval on the difference
−1.21 to −0.97 days
Cohen's d
−0.32 (small to medium)
Mann–Whitney check
p = 5.2 × 10⁻⁷⁰: agrees
Here the effect is both statistically overwhelming and practically meaningful: a full day off the working-capital cycle, with an interval tight enough that the size is not in doubt. The rank-based check is there because a reviewer will ask whether the mean is the right summary for skewed billing data, and being able to say that both tests agree closes that line of challenge before it opens.
Welch's standard error
Portal v2's 6,031 customers paid in 14.36 days on average with s = 3.40; the old portal's 5,969 in 15.44 days with s = 3.48. What is Welch's standard error of the gap, in days? Four decimals.
A decimal or a fraction; within 0.0001 of the answer counts as right.
What it computes
Python (scipy, statsmodels)
Excel
Welch's t test
ttest_ind(treatment, control, equal_var=False)
=T.TEST(B2:B6032, C2:C5970, 2, 3)
Levene's test, centred on the median
levene(treatment, control)
=F.DIST.RT(1.23, 1, 11998)
The Mann–Whitney check
mannwhitneyu(treatment, control)
=2*(1-NORM.S.DIST(ABS(B4), TRUE))
ttest_ind, levene and mannwhitneyu are in scipy.stats. Excel's T.TEST with type 3 is Welch's test. Excel has no Levene or Mann–Whitney function: the lines turn the statistic into a p-value (Levene's W on 1 and 11,998 degrees of freedom; Mann–Whitney's U as a normal z in B4, found with RANK.AVG and the formula).
Step 6: H3, the paired test, and what pairing buys you
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 5 are built, step 6, H3, calls per agent, is marked as the current one, and the rest are still to come.
H3: the same change per agent, analysed paired and as independent groups
Two intervals for the same average change, −3.89 calls per agent per day. The paired analysis gives a narrow interval, −5.11 to −2.66, far from zero. Analysed wrongly as two independent groups, the interval widens to −6.97 to −0.80 and nearly reaches zero: the spread between agents buries the effect.
Eighty call-centre agents were measured twice: average daily calls in the four weeks before the rollout and in the four weeks after. Because the same agent appears in both columns, the samples are not independent: an agent on a busy queue is busy in both periods.
Identical data, a different design assumption, and about six orders of magnitude between the p-values
Analysis
t statistic
p-value
Verdict
Paired t test (correct)
−6.32
1.5 × 10⁻⁸
Strong evidence
Independent t test (wrong)
−2.49
0.014
Weak evidence
Agents handle between about 31 and 85 calls a day. That spread between agents swamps a 4-call improvement when the columns are compared as independent samples. Computing each agent's own change cancels the between-agent variation, leaving only the variation in the change itself. The pairing was free, sitting in the data; failing to use it would have buried a real effect.
In words: \(\bar{d} = -3.89\) is the average change per agent, \(s_d = 5.50\) the standard deviation of the 80 changes and \(n = 80\), so \(t = -3.89 / (5.50/\sqrt{80}) = -6.32\) on 79 degrees of freedom. The mean improvement is 3.89 fewer calls per agent per day, 95 % interval 2.66 to 5.11. The before and after columns correlate at r = 0.85: that shared agent-to-agent spread is what the pairing removes.
The paired test needs the differences, not the raw levels, to be roughly normal. Their skewness is 0.46, not far from symmetric, and 80 pairs is a comfortable size for the mean of the differences. The tutorial also runs a Shapiro–Wilk test on them; its scipy line is below, and chapter 15 explains why the lab prefers a look at the data to that test.
The caveat that belongs in the readoutThis is a before-and-after comparison with no control group, so it is not causal evidence in the way H1 and H2 are. Seasonality, a concurrent tariff change or simply a quiet billing month would produce the same pattern. It is therefore reported as a supporting operational indicator, with that limitation stated. A statistically flawless test of a badly designed comparison is still a badly designed comparison: design outranks arithmetic.
Spot the mistake: three orders of magnitude
Spot the mistake
The same 80 agents give p = 1.5 × 10⁻⁸ with the paired test and p = 0.014 when analysed as independent groups. A report says the p-values are "three orders of magnitude apart". How many orders of magnitude apart are they, to the nearest whole number?
A whole number.
What it computes
Python (scipy, statsmodels)
Excel
The paired t test
ttest_rel(calls_post, calls_pre)
=T.TEST(C2:C81, B2:B81, 2, 1)
The same data as independent groups (the wrong design)
ttest_ind(calls_post, calls_pre, equal_var=False)
=T.TEST(C2:C81, B2:B81, 2, 3)
The shape of the differences
shapiro(calls_post - calls_pre)
=SKEW(D2:D81)
ttest_rel, ttest_ind and shapiro are in scipy.stats. In Excel, column D holds each agent's change, =C2-B2; Excel has no Shapiro–Wilk test, so the line looks at the skewness instead.
Step 7: H4, ANOVA across four regions, and why the omnibus is not enough
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 6 are built, step 7, H4, the regions, is marked as the current one, and the rest are still to come.
H4: Tukey's intervals for the six pairs of regions
Six intervals on a scale of days, one per pair of regions, each the difference in mean days to payment with its Tukey–Kramer 95 % interval, against a line at zero. Five intervals lie wholly on one side of zero. The North–West interval, −0.03 to 0.43, crosses it: that pair is not distinguishable. The three pairs with East show it paying 1.5 to 2.3 days later than the others.
Levene's test across the four regions gives p = 0.52, so the variances are comparable and the classic one-way ANOVA is valid. It compares the spread between the region means with the spread within the regions:
Region explains 6.5 % of the variance in days to payment, a medium effect by convention and operationally significant for a collections function (p = 1.4 × 10⁻¹⁷⁴). But F says only "at least one region differs". Tukey's HSD, which holds the family-wise error rate at 5 % across all six pairwise comparisons, says which. With regions of different sizes (2,365 to 3,600 customers) it is the Tukey–Kramer form, the one scipy's tukey_hsd computes (correction C34).
Tukey's HSD: five of six pairs differ; one does not
Comparison
Mean difference (days)
Significant?
North minus East
−2.35
Yes
South minus East
−1.46
Yes
West minus East
−2.15
Yes
South minus North
+0.89
Yes
West minus North
+0.20
No (p = 0.116)
West minus South
−0.69
Yes
Note the North–West row. The omnibus test was significant beyond any doubt, yet this pair is not distinguishable. A significant F never means every pair differs, a mistake that leads straight to regional action plans aimed at differences that do not exist. The actionable finding is narrower and more useful: East lags the other three regions by 1.5 to 2.3 days and warrants its own diagnosis.
The pair that does not differ
The ANOVA across Helion's four regions is overwhelmingly significant. Which pair of regions does Tukey's HSD not distinguish?
What it computes
Python (scipy, statsmodels)
Excel
One-way ANOVA
f_oneway(east, north, south, west)
=F.DIST.RT(278.2, 3, 11996)
Tukey's HSD (Tukey–Kramer for unequal groups)
tukey_hsd(east, north, south, west)
=ABS(B1-B2)/SQRT(B3/2*(1/B4+1/B5))
η², the share of variance explained
ss_between / ss_total
=B6/B7
f_oneway and tukey_hsd are in scipy.stats; Excel's Data Analysis ToolPak runs the ANOVA table ("Anova: Single Factor"). Excel has no studentized range distribution, so it can compute Tukey's q for a pair (the second line: two means, the mean square within and the two group sizes in B1 to B5) but not its p-value.
Step 8: H5, χ² independence, and reading beyond the p-value
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 7 are built, step 8, H5, segment and payment method, is marked as the current one, and the rest are still to come.
H5: where segment and payment method depart from independence
H5: where segment and payment method depart from independence
Segment
Bank Transfer
Card
Cash
Direct Debit
Corporate
+24.4 ●
570 observed, 213.2 expected
−10.4 ●
74 observed, 233.5 expected
−5.1 ●
12 observed, 46.5 expected
−7.4 ●
319 observed, 481.8 expected
Residential
−15.2 ●
1,179 observed, 1,829.1 expected
+6.6 ●
2,300 observed, 2,003.4 expected
+4.0 ●
478 observed, 398.7 expected
+4.3 ●
4,408 observed, 4,133.7 expected
SME
+12.2 ●
875 observed, 581.7 expected
−5.4 ●
500 observed, 637.1 expected
−4.0 ●
82 observed, 126.8 expected
−3.1 ●
1,203 observed, 1,314.5 expected
Each cell: the standardised residual, (observed − expected) / √expected. ● marks a cell beyond ±2, one that contributes materially; + means more than independence expects, − fewer.
A heat map of the 3 × 4 table of segment by payment method. Each cell shows its standardised residual, (observed − expected) / √expected, with its sign and number; cells beyond ±2 are outlined. Corporate × Bank Transfer stands out at +24.4; Residential leans towards Card, Cash and Direct Debit and away from Bank Transfer (−15.2).
A 3 × 4 table of customer segment against payment method. The smallest expected count is 46.5, comfortably above 5, so the χ² approximation is safe.
Take Corporate customers paying by bank transfer. There are 975 Corporate customers and 2,624 bank-transfer payers among 12,000, so independence expects \(E = 975 \times 2{,}624 / 12{,}000 = 213.2\) of them; 570 were observed. The cell's standardised residual is \((570 - 213.2)/\sqrt{213.2} = 24.4\). Summed over the 12 cells, χ² = 1,298 on (3 − 1)(4 − 1) = 6 degrees of freedom, p = 2.5 × 10⁻²⁷⁷, and Cramér's V = 0.233.
The association is real and of small-to-medium strength. But the omnibus statistic is not actionable on its own. The standardised residuals, with anything beyond ±2 marking a cell that contributes materially, show what the association is: Corporate customers concentrate heavily in bank transfer, while Residential customers concentrate in direct debit and card.
That is the finding a collections strategy can be built on. A single organisation-wide "move everyone to direct debit" campaign would be aimed at the wrong behaviour for a whole segment. The p-value could never have told anyone that; the residuals do.
An expected count
Helion has 8,365 Residential customers and 572 customers who pay in cash, out of 12,000. If segment and payment method were independent, how many Residential cash payers would you expect? Two decimals.
A decimal or a fraction; within 0.01 of the answer counts as right.
In Excel, B2:E4 hold the observed table with its totals in column F and row 5, and B7:E9 the expected table built from them; B11 holds χ². Neither Excel nor scipy's chi2_contingency applies Yates's correction to a table larger than 2 × 2.
Step 9: H6, goodness of fit, where failing to reject is the good news
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 8 are built, step 9, H6, the channel mix, is marked as the current one, and the rest are still to come.
H6: contacts by channel against the capacity plan's forecast
Four pairs of bars, one pair per channel: the contacts observed over the experiment window beside the counts the forecast's shares predict for the same 1,600 contacts. Phone 742 against 736, Chat 388 against 384, Email 271 against 288, Self-serve 199 against 192: every pair is close.
The quarter's capacity plan assumed a contact-channel mix of 46 % phone, 24 % chat, 18 % email and 12 % self-serve. Does the mix over the experiment window match it?
Goodness-of-fit contributions: email is the largest contributor, and even it is small
Channel
Observed
Expected
Contribution to χ²
Phone
742
736.0
0.049
Chat
388
384.0
0.042
Email
271
288.0
1.003
Self-serve
199
192.0
0.255
We fail to reject H₀. The observed mix is consistent with the forecast the staffing model was built on, so no re-plan is required. Note the wording carefully: this does not prove the forecast is correct, only that the data give no reason to abandon it. The distinction matters as much in a capacity meeting as in a statistics exam (chapter 3's "fail to reject").
Word a non-rejection
The channel-mix test gives χ² = 1.349 on 3 degrees of freedom, p = 0.717. Which sentence belongs in the capacity meeting?
Excel's CHISQ.TEST takes the observed and expected columns and uses k − 1 degrees of freedom by itself: right here, where nothing was estimated from the data.
Step 10: the family, because six tests at 5 % is not a 5 % error rate
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 9 are built, step 10, the family, is marked as the current one, and the rest are still to come.
The family: six p-values against the corrections
Six horizontal bars, one per hypothesis, each as long as −log₁₀ of its p-value, capped at 10 with an arrow for the smaller ones. A line marks α = 0.05 and another Bonferroni's α/6; a tick on each bar marks its Benjamini–Hochberg threshold. H1 to H5 reach far past every line; H6, at p = 0.717, reaches none of them.
Running six tests at α = 0.05 gives a family-wise false-positive probability well above 5 %: if every null were true and the tests independent, 1 − 0.95⁶ = 26 %. Two standard corrections answer it, and they promise different things:
Bonferroni against Benjamini–Hochberg
Method
Controls
Behaviour
When to use
Bonferroni
The family-wise error rate: the probability of any false positive
Divides α by the number of tests. Very conservative; costs real power
Confirmatory work, regulated settings, few tests
Benjamini–Hochberg
The false discovery rate: the expected share of the rejections that are false
Ranks the p-values and applies a graduated threshold. Less conservative
Exploratory analytics, where a few false leads are tolerable but missed effects are not
The family: Benjamini–Hochberg, Bonferroni and Holm on the six p-values
Hypothesis
p
BH adjusted
Bonferroni adjusted
Holm adjusted
Decision (BH)
H1
4.2 × 10⁻⁵
5.1 × 10⁻⁵
2.5 × 10⁻⁴
8.4 × 10⁻⁵
Reject H₀
H2
1.3 × 10⁻⁶⁶
2.6 × 10⁻⁶⁶
7.8 × 10⁻⁶⁶
5.2 × 10⁻⁶⁶
Reject H₀
H3
1.5 × 10⁻⁸
2.2 × 10⁻⁸
8.8 × 10⁻⁸
4.4 × 10⁻⁸
Reject H₀
H4
1.4 × 10⁻¹⁷⁴
4.2 × 10⁻¹⁷⁴
8.4 × 10⁻¹⁷⁴
7.0 × 10⁻¹⁷⁴
Reject H₀
H5
2.5 × 10⁻²⁷⁷
1.5 × 10⁻²⁷⁶
1.5 × 10⁻²⁷⁶
1.5 × 10⁻²⁷⁶
Reject H₀
H6
0.717
0.717
1.000
0.717
Fail to reject H₀
The notebook applies the pre-registered Benjamini–Hochberg correction and reports Bonferroni beside it. All three corrections agree on all six hypotheses: H1, H2, H3, H4, H5 are rejected and H6 is not. That is the comfortable case: the conclusions do not depend on the choice of correction.
When they disagree, which typically happens for a result between 0.01 and 0.05, say so explicitly rather than quoting whichever supports the recommendation. Reporting both is also what makes the pre-registration meaningful: an auditor can see the choice was not made after inspecting the results. And name what each promises: Benjamini–Hochberg keeps the expected share of false discoveries among the rejections at 5 %; Bonferroni and Holm keep the chance of even one false rejection at 5 % (correction C30).
Bonferroni by hand
H1's raw p-value is 4.22 × 10⁻⁵ and the family has six hypotheses. What is its Bonferroni-adjusted p-value? Give it to four decimals.
A decimal or a fraction; within 0.0001 of the answer counts as right.
What it computes
Python (scipy, statsmodels)
Excel
Benjamini–Hochberg (pre-registered)
multipletests(p, alpha=0.05, method='fdr_bh')
=MIN(1, B2*6/C2)
Bonferroni
multipletests(p, alpha=0.05, method='bonferroni')
=MIN(1, 6*B2)
Holm
multipletests(p, alpha=0.05, method='holm')
=MIN(1, (6-C2+1)*B2)
multipletests is in statsmodels.stats.multitest. In Excel, B2 holds a p-value and C2 its rank from the smallest; the BH and Holm lines also need the running minimum or maximum down the sorted column, which statsmodels applies for you.
Step 11: power, sizing the next experiment
Where this step sits in the pipeline
The pipeline's twelve stages as a row of numbered steps: steps 1 to 10 are built, step 11, power for the next experiment, is marked as the current one, and the rest are still to come.
Power to detect finance's 3-point lift, by customers per arm
A curve rising from low power at a few hundred customers per arm towards 1. A horizontal line marks the conventional 80 %. The curve crosses it at 3,993 customers per arm, marked; a second mark at 5,969, this experiment's smaller arm, sits above the line, at a power of 0.93.
Two questions, both feeding the next design: how many customers would we have needed, and what is the smallest effect we could have detected?
The sample size for the primary metric
Input
Value
Baseline adoption
34 %
Target lift (finance's threshold)
3 percentage points
Cohen's h for that lift
0.063
Required n per arm (α = 0.05, power 80 %)
3,993
Actual n per arm
5,969: adequately powered
In words: \(h\) is the gap between 34 % and 37 % on the arcsine scale, which makes differences between proportions comparable wherever they sit; \(z_{0.975} = 1.96\) is the two-sided cut-off at α = 0.05 and \(z_{0.80} = 0.84\) buys 80 % power. So \(n = 2 \times ((1.96 + 0.84) / 0.0627)^2 = 3992.1\): round up to 3,993 customers per arm.
For days to payment at 5,969 per arm, the minimum detectable effect at 80 % power is a Cohen's d of 0.051, or 0.18 days in the original units. The observed 1.09 days sits far above it.
This second number is what makes a null result interpretable. "We found no difference" is uninterpretable on its own. "We found no difference, and this design would have detected anything larger than 0.18 days" is a finding. Report the minimum detectable effect beside every non-significant result, and the file drawer stops being a black hole.
Planning, not observed powerThese powers are worked out for the effect finance cares about, 3 points, not for the effect this experiment happened to observe. Power computed at an observed effect only restates the p-value and adds no evidence (correction C13, chapter 17).
Customers for the next experiment
The next experiment must detect finance's 3-point lift from a 34 % baseline with 80 % power at α = 0.05, two-sided. How many customers per arm? A whole number.
NormalIndPower and TTestIndPower are in statsmodels.stats.power. The Excel lines use the normal approximation, which at these sizes agrees with statsmodels to the digits shown.
Five experiments to run on the simulator
The tutorial ends with five exercises for its notebook. Here each is a preset of the simulator at the top of the chapter, and the links below open it set up. Every run is seeded, so the numbers quoted here are the ones you will see.
The design as run, five seeds: the p-value moves by orders of magnitude, the interval's width hardly at all
Seed
Lift (points)
95 % interval (points)
p
1
6.94
5.22 to 8.66
3.3 × 10⁻¹⁵
2
6.04
4.32 to 7.75
5.9 × 10⁻¹²
3
5.73
4.01 to 7.45
6.7 × 10⁻¹¹
4
5.90
4.18 to 7.62
1.9 × 10⁻¹¹
5
4.53
2.80 to 6.25
2.8 × 10⁻⁷
Change the seed. Across five seeds the p-value runs from 2.8 × 10⁻⁷ to 3.3 × 10⁻¹⁵, while every interval is about 3.4 points wide. The interval is the stable number; the p-value is not.
Try thisCheck Helion's split in the chi-square explorer: 5,969 against 6,031 against a planned 50/50, the sample ratio mismatch check.
800 customers. With 400 per arm, the smallest lift this design catches 80 % of the time is 9.7 points. The true lift is 5.2 points, so the design has too little power for it, and whether it reaches significance is luck: in this run the lift is 11.3 points with p = 0.001 (significant this time). Read the minimum detectable lift before reading a null result.
Try thisShrink the experiment to 800 customers and read the minimum detectable lift beside the result.
A half-point lift. With a true lift of 0.5 points and 400,000 customers, the test rejects (p = 1.7 × 10⁻⁷), but Cohen's h is 0.017 and the whole interval, 0.49 to 1.08 points, lies below finance's 3. Rejecting is the right statistical call; shipping would be the wrong business call.
Try thisMake the true lift half a point and the experiment very large: significant, and not worth shipping.
A broken split. Send 45 % to the old portal and 55 % to Portal v2, as a broken assignment service would: 5,356 against 6,644 customers, SRM χ² = 138 and p = 6.4 × 10⁻³². The gate halts the readout before H1 is tested. This is what a broken assignment looks like from the analyst's side.
Try thisBreak the randomisation at 45/55 and watch the SRM gate stop the readout.
A seventh hypothesis. Add a seventh test with p = 0.03 to the family, and every adjusted p-value moves: H1's Bonferroni-adjusted p goes from 2.0 × 10⁻¹⁴ to 2.3 × 10⁻¹⁴ and its Benjamini–Hochberg one from 5.0 × 10⁻¹⁵ to 5.9 × 10⁻¹⁵. Each new test raises the bar for the others: think of the analyst who runs forty a quarter.
Try thisAdd a seventh hypothesis to the family and compare H1's adjusted p-values.
The notebook is the analysis. Turning it into a repeatable capability means separating the three things a notebook naturally mixes: the data contract, the statistical logic and the readout. The tutorial's reference architecture on Azure keeps the decision as queryable data, not as a slide:
A reference architecture: the decision is persisted as data, not as a slide
Stage
Service
Responsibility
Assignment
App Service → Event Hubs
Emits (customer_id, variant, assigned_at) when a customer is randomised
Landing
ADLS Gen2
Raw assignment and outcome events, immutable
Curation
Synapse Spark or a Fabric notebook
Joins assignments to outcomes at customer grain; writes Delta
Gates
The same notebook, failing fast
SRM and quality; a failure stops the run before any analysis
Analysis
A notebook parameterised by experiment_id
The statistical logic of this chapter
Results
A Delta table and SQL Managed Instance
analytics.experiment_results, append-only, versioned by code version
Presentation
Power BI
Reads the results table directly; nobody re-runs a notebook
Alerting
Log Analytics and a Teams webhook
Posts SRM failures and completed readouts
Persisting the result as a row per hypothesis, with the experiment id, the adjusted p-value, the decision, the time and the code version, lets the dashboard render without re-running anything and gives audit a record that ties a business decision to a specific statistic, on a specific dataset, on a specific date, from a specific version of the code. And test the statistical code against known answers: a divisor of n where n − 1 belongs shifts every result slightly and breaks nothing visibly.
Governance: the rules that matter more than the code
No peeking. Do not run the readout daily and stop when p first drops below 0.05. Repeated looks inflate the false-positive rate: checking a test ten times during collection pushes a nominal 5 % to about 19 %. Fix the sample size in advance, or use a sequential design built to be looked at.
One primary metric. Secondary metrics are for diagnosis and new hypotheses. Promoting one to primary after the real primary disappoints is the most common form of results-shopping in industry.
Effect size is mandatory. No result reaches a business audience without an interval and an explicit comparison with the decision threshold.
Segment analyses are exploratory unless pre-registered. Slicing a null result by region, then segment, then tenure until something turns significant will always succeed in the end: that is what a 5 % error rate applied twenty times means. What unplanned slicing finds is a hypothesis for the next experiment, never a conclusion from this one.
Publish null results. A file drawer full of failed experiments distorts what an organisation believes works, exactly as publication bias distorts a scientific literature.
Separate statistical significance from the business decision. The test says whether chance is a plausible explanation. Whether to spend €4m is a judgement about the interval, the cost, the risk and the strategy. Analysts who present the two separately get trusted.
Why it matters
A readout is where statistics meets money. The same data could have been reported as "significant, ship it, €1.27m a year", and eighteen months later the programme would have looked like a failure. The pipeline makes the honest version the easy one: the plan is fixed before the data, broken data stop the run, every result comes with its interval and its threshold, the family is corrected as promised, and the memo carries the range, underwritten at its lower end.
When this goes wrong
The readout is only as good as its design. It assumes the randomisation worked (the SRM check can say it did not, never that it did in every respect), that customers did not affect one another, and that one billing cycle is long enough: a new portal can win on novelty and fade. H3's before-and-after comparison has no control group, so it cannot be read causally. The €8.40 saving per customer is an input from the business case, not a finding, and the money range is only as good as it. And the data here are the notebook's synthetic customers: the method transfers to a real experiment, the numbers do not.
The pipeline's twelve stages as a row of numbered steps: steps 1 to 11 are built, step 12, the memo, is marked as the current one, and the rest are still to come.
The final step produces the artefact the business actually reads. Note what it contains and what it does not: no p-values, no test statistics, no degrees of freedom. The audience needs the decision, the size of the effect, the uncertainty around it and the caveats. The statistics are the evidence, not the message. Three checks first, one for each sentence the notebook's own memo got wrong.
Spot the mistake: East lags by 2.3 days
Spot the mistake
A draft memo says: "Regional variation: East lags the others by roughly 2.3 days." What should it say?
Which figure underwrites the business case?
The memo's financial sensitivity reads: lower bound €661,398, point estimate €1,267,564, upper bound €1,873,729. Which figure should the business case be underwritten at?
Spot the mistake: what Benjamini–Hochberg controls
Spot the mistake
A memo's caveats read: "Six hypotheses were tested; Benjamini–Hochberg holds the family-wise error rate at 5 %." What is wrong?
What it computes
Python (scipy, statsmodels)
Excel
The money at the interval's lower bound
4_200_000 * ci_low * 8.40
=4200000*B2*8.40
The memo's lines from the results table, with no p-values
The memo is written from the persisted results, never typed: B2 holds the interval's lower end for the lift, then the two arms' rates.
Decision memo
Portal v2 readout: EXP-2026-014-PORTALV2
Helion Energy · for the board · no p-values, by design
Recommendation
Proceed to a phased rollout, with the paperless business case re-underwritten at the lower bound of the confidence interval.
Primary findings
Paperless adoption
34.4 % → 38.0 %: a lift of 3.6 points, 95 % interval 1.9 to 5.3 points. The interval straddles the 3-point threshold finance set, so the benefit is real but its size is not yet settled.
Days to payment
15.4 → 14.4 days: 1.09 days faster, 95 % interval 0.97 to 1.21 days. Consistent across the mean-based and the rank-based tests.
Supporting findings
Call volume
Down 3.9 calls per agent per day (before and after, no control group: indicative, not causal).
Regional variation
Region explains 6.5 % of the variation in days to payment; East lags the other three regions by 1.5 to 2.3 days, depending on the region.
Collections strategy
Segment and payment method are clearly associated (Cramér's V = 0.23): one strategy for everyone will misfire by segment.
Capacity planning
The channel mix is consistent with the forecast; no re-plan is required.
Financial sensitivity (a year, 4.2 million customers)
At the interval's lower bound
€661,398
At the point estimate
€1,267,564
At the interval's upper bound
€1,873,729
Checks and caveats
Randomisation
The split, 5,969 against 6,031, is consistent with 50/50. The check is strict because it runs on every experiment, and at this size it would still catch a 52/48 split 86 % of the time.
Multiple testing
Six pre-registered hypotheses, corrected with Benjamini–Hochberg, which controls the false discovery rate, and checked with Bonferroni, which controls the family-wise error rate: the same five findings stand either way.
Next experiment
3,993 customers per arm detect finance's 3-point lift with 80 % power.
That financial sensitivity block is the whole document in three lines. The point estimate is €1.27m; the honest range is €0.66m to €1.87m. A readout that quoted only the point estimate would be presenting a number the data do not support to a precision the data cannot deliver, and eighteen months later, when the realised benefit lands at €0.8m, the analysis would look wrong when in fact it had been right all along and simply reported badly.
Remember:A readout that deserves a decision is a pipeline, not a p-value: commit the plan before the data, stop on broken data and on a broken split, give every result its interval and its threshold, correct the family as promised, and put the money at the interval's lower bound. Significance says the effect is real; the interval says whether it is big enough.