Cheat sheet

When to use it: When σ comes from a long record of the process, not from this sample. Reject H₀ when p ≤ α, or when z lies beyond the critical value.

Chapter 12's line A, 40 bottles (ml), in A2:A41.

line_a = np.array([506.9, 504.9, 492.6, 486.9, 516, 514.9, 500.2, 478.7, 520, 507.5, 503.4, 499.1, 496.2, 508, 487.7, 503.9, 498.8, 500.3, 498.5, 495.6, 509.9, 498.6, 491.4, 502.6, 505.9, 480.2, 500.6, 485.5, 510.3, 507.6, 497.7, 488.8, 494.5, 510.6, 504.1, 500, 504.5, 504.6, 509, 491.3])

When to use it: To report the range of values for μ the data support. A two-sided test at α rejects exactly the values outside the 1 − α interval (for the z and t tests of a mean).

When to use it: One mean against a claimed value when σ is estimated from the sample, which is almost always. Check the shape for small samples; the Wilcoxon signed-rank test is the fallback.

Chapter 12's line A, 40 bottles (ml), in A2:A41.

line_a = np.array([506.9, 504.9, 492.6, 486.9, 516, 514.9, 500.2, 478.7, 520, 507.5, 503.4, 499.1, 496.2, 508, 487.7, 503.9, 498.8, 500.3, 498.5, 495.6, 509.9, 498.6, 491.4, 502.6, 505.9, 480.2, 500.6, 485.5, 510.3, 507.6, 497.7, 488.8, 494.5, 510.6, 504.1, 500, 504.5, 504.6, 509, 491.3])

When to use it: Two separate groups of units. Use Welch's test by default: it does not assume equal variances and costs almost nothing when they are equal. scipy pools the variances unless you pass equal_var=False.

Chapter 14's settlement times in days: office A in A2:A16, office B in B2:B16.

a = np.array([6.6, 2.1, 26.4, 5.3, 3.7, 3.4, 11.8, 5, 20.5, 9.7, 3.2, 8.2, 7, 3.6, 3.4])
b = np.array([18.8, 9.4, 6.4, 78.2, 7.5, 6.8, 12, 6.9, 15.8, 18.8, 7.5, 29.8, 15.4, 8.6, 28.8])

When to use it: The same units measured twice, or matched pairs. It is a one-sample t test on the differences; the design, not the p-value, decides that the data are paired.

Helion's first 12 agents, average daily calls before (A2:A13) and after (B2:B13).

pre = np.array([50.1, 40.6, 62.7, 69.6, 63.9, 68.6, 60.7, 54.2, 59.2, 53.3, 85.2, 64.5])
post = np.array([36.5, 38.2, 59.4, 63.3, 58, 63.7, 64.6, 41.8, 45, 50, 76.5, 67.6])

When to use it: A yes/no outcome against a target rate. The z test needs n p₀ and n(1 − p₀) both at least 10; otherwise, or when in doubt, use the exact binomial test.

When to use it: Two groups' rates, the workhorse of A/B testing. The test uses the pooled rate p̂, because H₀ says the rates are equal; the interval for the gap uses each group's own rate.

When to use it: A 2 × 2 table with fewer than 10 successes or failures in a group. Given the table's totals, it adds the exact probabilities of every table at least as extreme as the observed one.

When to use it: Counts in categories against a mix fixed in advance: a forecast, a 50/50 split (the sample ratio mismatch check). Counts, never percentages; every expected count at least 5.

Helion's contacts by channel in A2:A5, and the forecast's expected counts (the shares times 1,600) in B2:B5.

observed = np.array([742, 388, 271, 199])
expected = np.array([736, 384, 288, 192])

When to use it: Two categorical variables on the same units: is one associated with the other? scipy applies Yates' correction to a 2 × 2 table by default; Excel's CHISQ.TEST never does.

H1's counts in A2:B3 (Portal v2 first; paperless, then not), and the expected counts built from the margins in D2:E3.

table = np.array([[2290, 3741], [2052, 3917]])
expected = np.array([[2182.2168333333334, 3848.7831666666666], [2159.7831666666666, 3809.2168333333334]])

When to use it: One group's spread against a specification. Valid only for normal data, and very sensitive to anything else.

Chapter 12's line B, 40 bottles (ml), in B2:B41.

line_b = np.array([514.1, 478.4, 496.2, 503.6, 483.4, 507.7, 482.1, 513.5, 502, 496.4, 516.5, 518.6, 490.6, 496.9, 501.4, 498.5, 497.4, 535.8, 470.9, 519.9, 514.9, 498.5, 483.9, 516.6, 510.2, 497.2, 510.3, 502, 493.3, 521.2, 506.1, 502.4, 512.1, 497.5, 488, 499.3, 489.2, 499.9, 499.6, 500.1])

When to use it: Are two groups equally consistent? The F test is exact only for normal data; Levene's test (centred on the median) is the practical tool, and Fligner–Killeen a rank-based second opinion.

Chapter 12's two filling lines, 40 bottles each (ml): line A in A2:A41, line B in B2:B41.

line_a = np.array([506.9, 504.9, 492.6, 486.9, 516, 514.9, 500.2, 478.7, 520, 507.5, 503.4, 499.1, 496.2, 508, 487.7, 503.9, 498.8, 500.3, 498.5, 495.6, 509.9, 498.6, 491.4, 502.6, 505.9, 480.2, 500.6, 485.5, 510.3, 507.6, 497.7, 488.8, 494.5, 510.6, 504.1, 500, 504.5, 504.6, 509, 491.3])
line_b = np.array([514.1, 478.4, 496.2, 503.6, 483.4, 507.7, 482.1, 513.5, 502, 496.4, 516.5, 518.6, 490.6, 496.9, 501.4, 498.5, 497.4, 535.8, 470.9, 519.9, 514.9, 498.5, 483.9, 516.6, 510.2, 497.2, 510.3, 502, 493.3, 521.2, 506.1, 502.4, 512.1, 497.5, 488, 499.3, 489.2, 499.9, 499.6, 500.1])

When to use it: Three or more groups' means at once, instead of many t tests. A significant F says some means differ; Tukey's HSD (the Tukey–Kramer form) says which.

A made-up example: delivery days at three depots, five parcels each, in A2:A6, B2:B6 and C2:C6.

d1 = np.array([2.9, 3.4, 3.1, 3.8, 3.3])
d2 = np.array([3.9, 4.4, 3.6, 4.1, 4.5])
d3 = np.array([3.2, 3, 3.7, 3.5, 2.8])

When to use it: When the data are skewed or ordinal and n is small. Each asks whether one group tends to give larger values, not whether the means differ.

The offices' settlement times (a, b), the 12 agents' calls (pre, post) and the three depots (d1, d2, d3), as in the entries above.

a = np.array([6.6, 2.1, 26.4, 5.3, 3.7, 3.4, 11.8, 5, 20.5, 9.7, 3.2, 8.2, 7, 3.6, 3.4])
b = np.array([18.8, 9.4, 6.4, 78.2, 7.5, 6.8, 12, 6.9, 15.8, 18.8, 7.5, 29.8, 15.4, 8.6, 28.8])
pre = np.array([50.1, 40.6, 62.7, 69.6, 63.9, 68.6, 60.7, 54.2, 59.2, 53.3, 85.2, 64.5])
post = np.array([36.5, 38.2, 59.4, 63.3, 58, 63.7, 64.6, 41.8, 45, 50, 76.5, 67.6])
d1 = np.array([2.9, 3.4, 3.1, 3.8, 3.3])
d2 = np.array([3.9, 4.4, 3.6, 4.1, 4.5])
d3 = np.array([3.2, 3, 3.7, 3.5, 2.8])

When to use it: Two numbers measured on the same units: do they move together? Pearson for a straight line; Spearman (Pearson on ranks) when the relationship is curved but one-directional, or there are outliers.

Helion's first 12 agents, average daily calls before (A2:A13) and after (B2:B13).

pre = np.array([50.1, 40.6, 62.7, 69.6, 63.9, 68.6, 60.7, 54.2, 59.2, 53.3, 85.2, 64.5])
post = np.array([36.5, 38.2, 59.4, 63.3, 58, 63.7, 64.6, 41.8, 45, 50, 76.5, 67.6])

When to use it: Before the test, for the smallest effect worth acting on; always round the sample size up. Never after it at the observed effect: that is a restatement of the p-value.

When to use it: Several tests on one experiment. Bonferroni and Holm control the chance of any false win (the family-wise error rate); Benjamini–Hochberg controls the false discovery rate. Pre-register which.

Four p-values from one experiment, sorted, in B2:B5.

p = np.array([0.001, 0.012, 0.03, 0.2])

When to use it: Counts of events over unequal time in service (claims per vehicle-year, incidents per machine-hour): compare the rates, not the share of units that had an event.

When to use it: Two models or raters scored on the same cases. Only the disagreements count: under H₀ each goes either way with probability ½. statsmodels' mcnemar is exact unless you pass exact=False.

When to use it: To show that a difference is small enough to ignore, within a margin fixed before the data and on the scale the decision uses. Not significant is not the same as equivalent.

Chapter 12's two filling lines, 40 bottles each (ml): line A in A2:A41, line B in B2:B41.

line_a = np.array([506.9, 504.9, 492.6, 486.9, 516, 514.9, 500.2, 478.7, 520, 507.5, 503.4, 499.1, 496.2, 508, 487.7, 503.9, 498.8, 500.3, 498.5, 495.6, 509.9, 498.6, 491.4, 502.6, 505.9, 480.2, 500.6, 485.5, 510.3, 507.6, 497.7, 488.8, 494.5, 510.6, 504.1, 500, 504.5, 504.6, 509, 491.3])
line_b = np.array([514.1, 478.4, 496.2, 503.6, 483.4, 507.7, 482.1, 513.5, 502, 496.4, 516.5, 518.6, 490.6, 496.9, 501.4, 498.5, 497.4, 535.8, 470.9, 519.9, 514.9, 498.5, 483.9, 516.6, 510.2, 497.2, 510.3, 502, 493.3, 521.2, 506.1, 502.4, 512.1, 497.5, 488, 499.3, 489.2, 499.9, 499.6, 500.1])