When the assumptions fail: ranks and permutations

What you just saw

Two claims offices each settled 15 claims. Office A took 8.0 days on average and office B 18.0: a gap of 10.05 days. But look at the dots. Most claims at both offices settle within two or three weeks, and a few take far longer, one at office B 78.2 days. The data are heavily skewed and the samples are small: exactly the case where a t test's normal shape is least safe.

So the animation built the null world from the data themselves. If the office made no difference, the labels "A" and "B" would be arbitrary: any 15 of the 30 times could have been office A's. Shuffling the labels 4,999 times and recording each shuffle's difference gives the differences luck alone produces. The real one, −10.05 days, lands out in the tail: only 51 shuffles were as low, and the permutation test's two-sided p-value is 0.0208.

The histogram has two humps. Shuffles that put the 78.2-day claim in office A land on the right; those that put it in office B land on the left. One claim that slow shapes the whole null world, a warning that the mean is a fragile summary of these data.

The permutation test: a null world from shuffles

A permutation test needs no formula for the null distribution and no assumption about its shape. It needs one thing: that under H₀ the labels are interchangeable, so every relabelling was as likely as the one we saw. Then:

In words: \(D\) is the real difference in means (office A minus office B, −10.05 days), \(D^{*}\) a shuffled one, and \(R\) the number of shuffles (4,999). The 1 added on top and bottom counts the real labelling as one of the possible ones, which keeps p above 0; scipy does the same. For the offices, 51 shuffles came out at −10.05 or lower, so the lower tail's p is \((1 + 51) / (1 + 4999) = 0.0104\), and doubling the smaller tail for a two-sided test gives 0.0208.

There are 155,117,520 ways to pick which 15 of the 30 times are office A's, too many to list, so the test samples 4,999 of them at random. With a seed, the same shuffles come out every time.

Compare Welch's t test on the same 30 numbers: t = −1.99 on about 17.9 degrees of freedom, p = 0.062, not significant at 5 %. Welch's p leans on the sample means being close to normal, which 15 skewed values do not give; the permutation test does not lean on it. That does not make the smaller p-value the right one to report. The test for small, skewed samples is chosen before the data are seen, not after comparing p-values.

Rank tests: when only the order counts

Parametric tests assume a distributional shape. When that assumption fails badly (small samples with heavy skew, ordinal outcomes such as satisfaction ratings, or a handful of extreme outliers), use the rank-based equivalent. These tests replace every value by its rank, so the slowest claim counts as rank 30 whether it took 30 days or 300. They ask whether one group tends to give larger values than the other, rather than whether the means differ, and they trade a little power for freedom from the shape assumption.

The Mann–Whitney U test is the rank version of the two-sample t test. Rank all 30 values together, add up office A's ranks (\(R_1\)), and subtract the smallest rank sum A could have had:

For the offices: office A's ranks add up to \(R_1 = 167\), so \(U = 167 - 15 \times 16 / 2 = 47\). If the offices did not differ, U would average \(15 \times 15 / 2 = 112.5\), with a standard deviation \(\sigma_U\) of 24.10 (a small correction for tied values included). So \(z = (|47 - 112.5| - 0.5) / 24.10 = 2.70\), and the two-sided p-value is 0.0070: office B's claims tend to take longer.

The Wilcoxon signed-rank test does for one sample or paired differences what Mann–Whitney does for two groups: it ranks the differences' sizes and compares the ranks of the positive and the negative ones.

With thousands of rows, do I still need a rank test?

Helion's days to payment are right-skewed, but each arm has about 6,000 customers. Welch's t test gives p = 1.3 × 10⁻⁶⁶ and Mann–Whitney p = 5.2 × 10⁻⁷⁰: both far below 0.05, so both reject. The lab computes that agreement from the two p-values rather than printing it as a fixed sentence, because a sentence typed in advance says "both agree" even on the day they do not.

Three tests, three questions

Meridian Assurance wants to know whether telematics changed the cost of a claim. Its claim amounts are extremely skewed (skewness 6.14 on the raw scale, 0.05 after taking logs), and three reasonable tests give three different answers:

• Welch on the raw amounts asks whether the arithmetic means differ: £4,730 against £5,964, p = 4.3 × 10⁻⁵. The mean is what a tariff pays out in total.
• Welch on the log amounts asks whether the geometric means, the typical claim on a multiplicative scale, differ: £2,067 against £2,223, a ratio of 0.930, p = 0.069.
• Mann–Whitney asks whether a telematics claim tends to be smaller than a standard one: the medians are £2,101 and £2,149, p = 0.148.

None of the three is wrong. The typical claim barely changed; a few very large standard claims make the standard tier's average much higher. Which answer matters depends on the decision: pricing runs on the arithmetic mean. Choose the test by the question, before looking at the data, and say which question it answered.

Several groups: Kruskal–Wallis, then pairs with Holm

The Kruskal–Wallis test is the rank version of one-way ANOVA. Meridian's claims in its four regions give H = 42.6 on 3 degrees of freedom, p = 3.0 × 10⁻⁹: at least one region's claims tend to be larger. How much larger? Its effect size, epsilon-squared, is small:

Region accounts for under 1 % of the variation in claim ranks: real, and small. As after ANOVA, the next question is which regions differ. Six Mann–Whitney tests, one per pair, are corrected with Holm's method (chapter 18) to keep the family-wise error rate at 5 %:

Three of the six pairs differ, and all three involve London, whose claims tend to be the largest. The other regions cannot be told apart.

Why it matters

A service-level agreement is often judged on a handful of cases, and case times are almost always skewed. A t test on 15 skewed values can miss a real difference or, with an outlier on the other side, invent one. Deciding in advance to use a permutation or a rank test, and saying which question it answers, makes the verdict one a reviewer can trust.

When this goes wrong

Rank and permutation tests are not assumption-free. They assume independent observations, like every test here. A permutation test of a difference in means assumes the two groups would be interchangeable under H₀, which is stronger than "same mean": if one office's times are simply more spread out, the test can reject with equal means. Mann–Whitney tests whether one group tends to give larger values, not whether the medians differ, unless the two shapes are the same. And a rank test says nothing about the size of the effect in days or pounds: report the difference and its interval too.

In code

permutation_test, mannwhitneyu, ttest_ind and kruskal are in scipy.stats; multipletests is in statsmodels.stats.multitest. Excel has no permutation or rank test: the Excel lines give the observed difference, one value's rank and Welch's p, and Holm's adjustment for one p-value (it must then be made non-decreasing down the sorted list). The settlement times are in the lab's code, seeded; Meridian's claims are Charles's notebook's own data.