Hypothesis Testing Lab

Twelve invoices took 5.8 days on average against a promise of 5. If the promise held, t would follow this curve; the shaded tails show how often a result at least this far out would happen by luck alone. That share is the p-value.

An interactive course in deciding from data: from the null hypothesis to p-values, errors and power, the everyday tests and the traps most courses skip. Every number comes from the lab's own engine, tested against scipy and statsmodels, and is shown three ways: the picture, the formula and the code.

Start the course Open the t-test explorer Refresh the t distribution

The whole course is open: twenty chapters in six parts, each with its animations, quiz and exercises, a test explorer for every family of tests, the tools, the cheat sheet and the lecture paths.

Part A: The logic of a test

The null world

How do I know a difference isn't just luck? Build the world where nothing changed and see where the real result lands.

Hypotheses and tails

What exactly am I testing, and in which direction? The null and the alternative, one tail or two, chosen before the data.

The p-value

What does p = 0.046 actually mean? Signal over noise, the shaded tail, the decision rule and four misreadings.

Errors and power

How often will a test fool me, and how often will it miss? Type I and II errors, power and its four levers.

Tests and confidence intervals

Why do people report an interval instead? One calculation seen two ways, and where the two can disagree.

Part B: Means

One mean: z and t, one complete test

σ is never known. What changes? The t test and the tutorial's twelve steps, end to end.

Two independent means

Did the new portal get people paying sooner? Welch's t test by default, and when pooling is justified.

Paired samples

Same people, before and after: why is that a different test, and what does pairing buy?

Part C: Proportions and counts

One proportion

Is 94 % on time below the 95 % promised? The z test for a proportion and the exact binomial test.

Two proportions

Did more customers go paperless? The two-proportion z test, its interval and Fisher's exact test.

Chi-square: goodness of fit and independence

Does the mix match the forecast? Does one category depend on another? Observed against expected counts.

Part D: Spread, several groups, and choosing

Variances: χ², F and Levene

Is a line more erratic than the spec allows? The χ² and F tests, and why Levene exists.

One-way ANOVA and after

Do several groups differ, and which ones? Between against within, and Tukey's intervals.

When the assumptions fail: ranks and permutations

Skewed data and a small sample: rank tests, the permutation test, and which question each answers.

Which test? The map, the assumptions, and correlation

Four questions that lead to the right test, the assumptions at a glance, and tests of correlation.

Part E: Using tests well

Effect size, intervals and "not significant"

It's significant: is it big? It's not significant: is it nothing? Effects, intervals and the smallest effect that matters.

Power and sample size

How many do I need before I start? Plan the sample for the smallest effect worth acting on.

Multiple testing, pre-registration and p-hacking

Twenty metrics and one came up: did we find something? Forking paths, peeking and the corrections.

Part F: Two readouts, end to end

An A/B test end to end: the Helion readout

Run one properly and tell the board: pre-registration, gates, six hypotheses and the decision memo.

A risk review end to end: the Meridian finale

Counts over exposure, the same cases scored twice, and a difference small enough to ignore.

Explorers

Calculators with sliders and typed inputs whose pictures move as you do: the statistic on its null distribution, the p-value shaded, the worked steps and the code.

The one-sample t test

Is a mean different from a promised value? The t statistic on its curve, with the p-value shaded as you move the data.

Two means: Welch, pooled and paired t tests

Did the new portal get people paying sooner? Compare two means with Welch's t test, the pooled test or the paired test, and watch the interval for the gap.

Proportions: one and two, z and exact

Is 94 % on time below the 95 % promised? Did the new portal convert more customers? Test one or two proportions with the z test or an exact test.

Chi-square tests: goodness of fit and independence

Does the contact mix match the forecast? Does payment method depend on segment? Type the counts and see each cell's share of χ².

Variances: χ², F, Levene and Fligner–Killeen

Is this filling line more erratic than the spec allows? Is one book riskier than another? Test a spread against a spec, or two and more spreads against each other.

One-way ANOVA with Tukey's test

Does days-to-payment differ by region, and which regions? One-way ANOVA from each group's summary, then Tukey's intervals for every pair.

Rank tests: Mann–Whitney, Wilcoxon and Kruskal–Wallis

My data are skewed and n is small. Now what? Compare groups by their ranks, with the exact distribution when the samples are small.

Correlation tests: Pearson and Spearman

Do two numbers move together, or is the pattern luck? Test Pearson's r or Spearman's ρ and see the interval for the correlation.

Rates over exposure: the Poisson rate ratio

Claims per vehicle-year, incidents per machine-hour: compare two rates when the groups had different exposure, with the score, Wald or exact test.

Paired proportions: McNemar's test

Two models scored on the same cases: is the challenger really more often right? McNemar's test looks only at the cases where they disagree.

Equivalence testing (TOST)

"Not significant" cannot say "no material change". An equivalence test can: two one-sided tests against a margin you set in advance.

Embed the explorers

Put the explorers on your own site, blog or course page with one line of HTML, free and without cookies.

Tools

Which test? The test chooser

Answer four questions about your data and watch the path light up to the right test, with its assumptions, its Python and its Excel.

Power and sample size calculator

How many do you need before you start? Solve for the sample size, the power or the smallest detectable effect, and see the power curve.

Multiple testing simulator: forking paths and peeking

Run an experiment where nothing works, slice it many ways, peek at it daily, and watch false wins appear. Then apply the corrections.

Reference

For looking things up and for teaching: every term with a link to where it is explained, every test with its formula and code, and lecture paths with speaker notes.

Glossary

80 terms from the chapters and tools, each linked to the section that explains it.

Cheat sheet

Every test on one printable page: the formula, when to use it, and the scipy, statsmodels and Excel that run it.

Lectures

4 guided routes through the chapters, figures and tools for a teaching session, with speaker notes at every step.

Where this lab starts

This lab picks up where the Introduction to Probability Lab ends. It does not repeat sampling distributions, the standard error, Student's t or confidence intervals: it links back to them.

Student's t and confidence intervals

Chapter 11 of the Introduction to Probability Lab: point estimates, the t distribution and the confidence interval that a two-tailed t test mirrors.

Open chapter 11 →

The Student's t explorer

The t distribution itself, in the Introduction to Probability Lab: its probabilities and critical values for any degrees of freedom, beside the normal.

An unhandled error has occurred. Reload 🗙