Which test? The map, the assumptions, and correlation
Four questions that lead to the right test, the assumptions at a glance, and tests of correlation.
A decision tree starts from one question on the left, what kind of outcome was measured, and branches to the right through how many groups, independent or paired, and whether σ is known, ending at 17 tests on the right. Five scenarios from chapters 6 to 13 go through it in turn: the bottles, Helion's days to payment, the agents' calls before and after, paperless adoption, and days to payment by region. For each, the path lights up one question at a time and the leaf names the test.
What you just saw
With the logic in place, the tests become a lookup exercise. The animation ran five questions from the course back through one decision tree: the bottles (one mean, σ unknown: the one-sample t test), Helion's days to payment (two independent groups: Welch's t test), the agents' calls before and after (paired: the paired t test), paperless adoption (yes or no in two groups: the two-proportion z test) and days to payment by region (four groups: ANOVA).
Each took a different path, and each path asked the same few questions in the same order. Four of them do most of the work, and they are the tutorial's: what kind of outcome, how many groups, independent or paired, and is σ known. The tree adds the questions the rest of the course needs: counts over exposure, the same cases scored twice, a margin for "no material difference", the spread of a number, and two numbers on the same units.
The four questions
Ask these four questions in order. They resolve nearly every case you will meet.
Question 1: what kind of outcome variable do I have? The outcome type does most of the work in choosing a test.
| Type | Examples | You will be testing | Test family |
|---|---|---|---|
| Numerical (continuous) | height, revenue, days to payment, response time | means | z, t, ANOVA |
| Categorical (counts) | yes or no, product A, B or C, churned or retained | proportions or counts | z for proportions, chi-square |
| Two numerical variables | height and weight; spend and tenure | association or prediction | correlation, regression |
Question 2: how many groups am I comparing? It sets the kind of test:
- One group against a fixed benchmark: a one-sample test.
- Two independent groups (different people in each): a two-sample test.
- Two measurements on the same units (before and after, matched pairs): a paired test.
- Three or more groups: ANOVA for a number, chi-square for a category.
Question 3: independent or paired? This one is decided by the study design, not the data, and getting it wrong is a real error rather than a stylistic one. Ask: does each observation in group 1 have a natural partner in group 2? Men's heights against women's heights have no partners: independent. Each employee's productivity before and after training has the same employee twice: paired. Patients matched on age and sex, one in each arm, are deliberately partnered: paired. Treating paired data as independent throws away the pairing and badly reduces power. Treating independent data as paired is worse: it is simply invalid.
Question 4: do I know the population standard deviation, σ? Almost never, in practice. If σ is really known (rare: a long-established process with historical data), use a z test. If you estimate it from the sample with s, use a t test. The t distribution is a slightly flatter bell with heavier tails than the normal, which is exactly the correction for having estimated σ. As n grows, s settles on σ and the t distribution approaches the normal; by around n = 30 they are close enough that the distinction rarely changes a decision. Software defaults to t, and that default is correct.
The complete map
Here are the tutorial's answers to the four questions, test by test, with the Python that runs each. The test chooser holds the same map as a tree, and adds the capstones' tests: the Poisson rate ratio for counts over exposure, McNemar's test for the same cases scored twice, and the equivalence test for "no material difference".
| Situation | Test | Python |
|---|---|---|
| One mean against a benchmark, σ known | One-sample z test | 2 * norm.sf(abs(z)) |
| One mean against a benchmark, σ unknown | One-sample t test | ttest_1samp(x, mu0) |
| Two independent means | Welch's t test (Student's only as a special case) | ttest_ind(a, b, equal_var=False) |
| Two paired means | Paired t test | ttest_rel(after, before) |
| Three or more means | One-way ANOVA | f_oneway(*groups) |
| Which groups differ, after ANOVA | Tukey's HSD | tukey_hsd(*groups) |
| One proportion against a benchmark | One-proportion z test, or exact binomial | binomtest(k, n, p0) |
| Two proportions | Two-proportion z test | proportions_ztest([k1, k2], [n1, n2]) |
| Observed counts against an expected mix | Chi-square goodness of fit | chisquare(observed, f_exp=expected) |
| Two categorical variables | Chi-square test of independence | chi2_contingency(table) |
| A sparse table (expected counts below 5) | Fisher's exact test | fisher_exact(table) |
| Two variances | Levene's test (the F test only for normal data) | levene(a, b, center='median') |
| A straight-line association between two numbers | Pearson correlation test | pearsonr(x, y) |
| A one-directional association, non-normal data | Spearman correlation test | spearmanr(x, y) |
| Predicting a number | Regression (t tests on the coefficients) | sm.OLS(y, X).fit() |
Assumptions at a glance
One-sample t test
- Assumes
- Independent observations; the sample mean roughly normal (normal data, or enough data for its skew).
- How to check
- A histogram or a Q-Q plot; look for extreme outliers.
- If it fails
- The Wilcoxon signed-rank test (chapter 14).
Welch's two-sample t test
- Assumes
- Independent groups and observations; each group's mean roughly normal. Equal variances are not needed.
- How to check
- Plot both groups; Levene's test only as information, never to choose the test.
- If it fails
- Mann–Whitney or a permutation test (chapter 14).
Paired t test
- Assumes
- Pairs independent of each other; the differences roughly normal (not the raw values).
- How to check
- Plot the differences.
- If it fails
- The Wilcoxon signed-rank test (chapter 14).
One-way ANOVA, then Tukey's HSD
- Assumes
- Independent observations; roughly normal residuals; similar variances across groups.
- How to check
- Levene's test; a plot of the residuals.
- If it fails
- Welch's ANOVA, or Kruskal–Wallis with pairwise Mann–Whitney and Holm (chapter 14).
Two-proportion z test
- Assumes
- Independent units; at least 10 successes and 10 failures in each group.
- How to check
- Count them in each group.
- If it fails
- Fisher's exact test.
Chi-square test of independence
- Assumes
- One row per unit (never the same unit twice); expected counts of at least 5.
- How to check
- Inspect the table of expected counts.
- If it fails
- Collapse categories, or Fisher's exact test.
Pearson correlation test
- Assumes
- A straight-line relationship; roughly normal; no strong outliers.
- How to check
- A scatter plot: always plot it.
- If it fails
- Spearman's rank correlation.
A grid of seven cards, one per everyday test: the one-sample t test, Welch's two-sample t test, the paired t test, one-way ANOVA, the two-proportion z test, the chi-square test of independence and the Pearson correlation test. Each card lists what the test assumes, how to check it, and the fallback if it fails, and links to that test's leaf in the test chooser.
One assumption deserves emphasis because it is invisible to every diagnostic in the grid: independence. If the same customer appears in your dataset three times, or if measurements cluster within stores, teams or households, your effective sample size is smaller than your row count. Every standard error is then too small, every p-value too small, and no normality test will hint at the problem. Independence is established by the design and the data grain, not by a test, which is why a careful readout starts by checking the grain (one row per customer) before it computes anything.
A note on "n ≥ 30". It is a rule of thumb for when the sample mean is close enough to normal, not a law, and it is about the mean's sampling distribution, never about the data. For strongly skewed data, such as claim amounts, 30 is far too few; for data that are already close to normal, fewer will do. Look at the data, then think about n.
When both variables are numbers: correlation
When both variables are numerical, the question shifts from "do the groups differ?" to "do the variables move together?". Helion measured 80 call-centre agents before and after the rollout. Do agents who took many calls before also take many after?
On ranks: Spearman's ρ = 0.85, t = 14.51 on 78 degrees of freedom, p = 7.2 × 10⁻²⁴. Pearson's r on the values is 0.85.
A scatter plot with one dot per agent: calls a day in the four weeks before the rollout across, in the four weeks after up. The dots rise together from lower left to upper right in a tight band, with a straight line through them. Pearson's r is 0.85. A switch replaces each value by its rank, 1 to 80, which is what Spearman's correlation uses: the picture barely changes and ρ is 0.85 too.
Pearson's correlation coefficient r measures how closely the points follow a straight line, from −1 to 1. Its test of H₀: ρ = 0 (no linear association in the population) is a t test:
In words: \(r\) is the sample correlation, \(n\) the number of pairs, and the t statistic grows with both. For the agents, r = 0.85 with n = 80, so \(t = 0.85 \times \sqrt{78} / \sqrt{1 - 0.85^2}\), which is 14.10 when r is kept to full precision, on 78 degrees of freedom: p = 3.8 × 10⁻²³. The 95 % interval for ρ, from Fisher's z transform, runs from 0.77 to 0.90.
Spearman's rank correlation ρ is Pearson's r computed on the ranks instead of the values. It asks only whether the variables move in the same direction, so it is robust to skew and outliers. For the agents it is 0.85 too: the relationship is a straight line and nothing is distorting it.
This correlation is why pairing helped in chapter 8. Busy agents are busy in both periods, so the before and after columns vary together. Their standard deviations are 9.45 and 10.26 calls a day, but the differences, each agent's own change, vary with a standard deviation of only 5.50: the between-agent spread cancels out.
| Test | H₀ | Assumes | Python |
|---|---|---|---|
| Pearson correlation | ρ = 0 (no linear association) | A straight-line relationship, roughly normal, no strong outliers | stats.pearsonr(x, y) |
| Spearman correlation | No monotonic association | Only that the relationship goes one way; robust to outliers and skew | stats.spearmanr(x, y) |
| Regression coefficient | β = 0 (this predictor adds nothing) | Linearity, independent errors, constant variance, normal residuals | sm.OLS(y, X).fit() |
Regression is worth recognising as hypothesis testing in disguise. Every coefficient in a regression summary comes with a t statistic and a p-value, testing H₀: β = 0, "this predictor contributes nothing once the others are accounted for". The overall F statistic tests whether the model as a whole beats an intercept-only model. Same logic, same decision rule, more moving parts. Regression itself is the subject of the Regression and Modelling Lab, planned for April 2027.
Why this is true: Why pairing shrinks the spread
The variance of a difference of two variables is the sum of their variances minus twice their covariance: \(s_d^2 = s_1^2 + s_2^2 - 2 r s_1 s_2\).
When the two columns move together (r close to 1), the last term cancels most of the first two. For the agents that leaves a standard deviation of 5.50 calls, against the 14.0 or so the two columns would give if they were unrelated. The paired test divides by the smaller number, which is where its power comes from.
Why it matters
The wrong test is the costliest mistake in this lab, because nothing warns you: every test returns a p-value. Analysing paired data as independent buried Helion's call reduction under the between-agent spread; testing a skewed mean with a rank test answers a question nobody asked. Four questions, asked before the data are opened, prevent both.
When this goes wrong
The tree is a map, not a substitute for looking at the data and the design. It cannot see clustering, duplicated rows or a before-and-after comparison with no control group; it cannot tell whether the mean is the right summary. A correlation, however significant, is not causation: the agents' before and after calls move together because the same agents took them, and a third factor (a busy queue) can drive both. And r only sees straight lines: a strong curved relationship can have r near 0, which is why the grid says always plot it.
In code
| What it computes | Python (scipy, statsmodels) | Excel |
|---|---|---|
| Pearson correlation test | pearsonr(calls_pre, calls_post) |
=T.DIST.2T(ABS(CORREL(B2:B81, C2:C81)*SQRT(78)/SQRT(1-CORREL(B2:B81, C2:C81)^2)), 78) |
| Its 95 % interval (Fisher's z) | pearsonr(calls_pre, calls_post).confidence_interval(0.95) |
=TANH(FISHER(CORREL(B2:B81, C2:C81))+NORM.S.INV(0.975)/SQRT(77)) |
| Spearman rank correlation | spearmanr(calls_pre, calls_post) |
=CORREL(D2:D81, E2:E81) |
| The correlation itself | np.corrcoef(calls_pre, calls_post)[0, 1] |
=CORREL(B2:B81, C2:C81) |
pearsonr and spearmanr are in scipy.stats. In Excel, the upper end of the interval is shown (subtract instead of add for the lower); for Spearman, first put each column's ranks in columns D and E with RANK.AVG. The agents' data are helion-agent-calls.csv: calls_pre in column B, calls_post in column C.
Quiz
Three questions on the chapter. Your best score is kept in this browser.