Variances: χ², F and Levene

What you just saw

Every test so far has been about a centre: a mean, a proportion. Sometimes the question is about spread. A filling line whose average is perfect but whose bottles swing wildly still overfills some and underfills others, and an insurer whose claims swing wildly needs more capital even if the average claim is the same. To test a spread we need to know how a sample's spread behaves by chance, and that is what the animation built.

First it took 400 samples of 10 bottles from a normal line with σ = 10 ml and worked out (n − 1)s²/σ² for each. The histogram filled the χ² curve with 9 degrees of freedom: 16 of the 400, about 1 in 20, landed beyond the 5 % critical value of 16.92, as promised. Then it took the ratio of two such samples' variances, and those ratios filled the F curve. Finally it repeated the first run on a skewed line, with the same σ but occasional big overfills: the histogram no longer matched the curve, and 54 of the 400 samples passed the critical value, 13.5 % against the promised 5 %.

That last panel is the whole story of this chapter. The χ² and F tests of spread are exact when the data are normal and unreliable when they are not, and that is why Levene's test exists.

Is this line more erratic than the spec allows?

Two lines fill the tutorial's 500 ml bottles. The spec says single bottles should vary with σ of no more than 10 ml. Line B has just come back from maintenance, and 40 of its bottles have s = 13.03 ml; line A's 40 have s = 9.38 ml. Is line B really more erratic than the spec, or is 13.03 the kind of s a line at σ = 10 produces by chance?

If the fills are normal, a sample's variance, scaled by the claimed one, follows the χ² curve with n − 1 degrees of freedom. That gives the test statistic:

In words: \(s^2\) is the sample's variance, \(\sigma_0^2\) the variance H₀ claims and \(n\) the sample size. If the line is exactly on spec, \(s^2\) is near \(\sigma_0^2\) and χ² is near its degrees of freedom. For line B, testing H₀: σ = 10 against Hₐ: σ > 10:

The upper-tail critical value at α = 0.05 is 54.57, and 66.22 is well beyond it: p = 0.0042. Line B is more erratic than the spec allows. Line A gives χ² = 34.29, p = 0.68: nothing to see. This test is exact only when the fills are normal, and very sensitive when they are not: unlike the t test, a large sample does not rescue it, because the central limit theorem is about means, not variances.

Are two lines equally consistent? The F test

To compare two spreads directly, divide one sample variance by the other. If the two lines are equally consistent and normal, the ratio follows the F curve with (n₁ − 1, n₂ − 1) degrees of freedom, the second curve in the animation. This is the F test:

F = 1.93, a two-sided p-value of 0.043: at 5 %, line B's spread is larger than line A's. Textbooks put the larger variance on top; that is a convention for reading upper-tail tables, not an assumption of the test. Either way round, the two-sided p-value is the same.

Levene's test: the practical tool

Levene's test asks the same question without leaning on normality. It works out each bottle's distance from its own line's centre and runs a one-way ANOVA (chapter 13) on those distances: if one line's bottles sit further from their centre, its spread is larger. Centred on the median, which is scipy's default and the robust choice, it is sometimes called the Brown–Forsythe test.

On the two lines Levene gives W = 2.29, p = 0.134: not significant. The F test said 0.043. Which is right? Here both are behaving as designed. These two lines are normal (they were generated that way), so the F test's assumption holds and it uses the data fully; Levene gives up some power for protection the lines do not need. A rank-based second opinion, the Fligner–Killeen test, agrees with Levene (p = 0.186). The honest reading is "line B looks more erratic, the evidence is moderate, and it depends on normality": take more bottles from line B before re-tuning it. The two lines' means, for the record, are indistinguishable (Welch, p = 0.64).

In practice you rarely know that your data are normal, and the last panel of the animation shows what the χ² and F tests do when they are not: false alarms at more than twice the promised rate. That is why Levene is the default tool for spread, and why chapter 7's Welch test never needs a variance pre-test at all.

Spread as the question: Meridian's claims

Spread is usually demoted to an assumption check. For an insurer it is a question in its own right. Capital requirements (in Europe set by Solvency II, the rules that decide how much capital an insurer must hold against bad years) are driven by the tail of the loss distribution, not its mean. Two books with the same expected loss but different spread do not consume the same capital or pay the same for reinsurance. A credible claim that telematics reduces the swing in claim sizes is worth money even if the average claim does not change.

Meridian Assurance's telematics pilot gives two groups of settled claims: 2,062 on telematics policies and 2,601 on standard ones. Claim sizes are heavily skewed, from a windscreen to a serious injury, so the question is asked on the log scale, where a £200 claim and a £2 million one are within reach of each other. The standard deviation of log severity is 1.30 for telematics and 1.41 for standard: a variance ratio of 1.17.

The classical F test is the wrong tool here: even on the log scale, claim sizes have heavier tails than a normal curve. Levene's test, median-centred, stays valid under exactly this kind of skew: W = 18.29, p = 1.9 × 10⁻⁵. Fligner–Killeen, as a second opinion, agrees: p = 1.8 × 10⁻⁵. Telematics claims are genuinely less dispersed. The classical F test also says the spreads differ (p = 0.00014), but on data like these its p-value cannot be trusted.

The table shows why capital cares. The typical claim is the same on both products; the 99th percentile, the kind of claim that drives capital and reinsurance, is 30 % lower on telematics. The test says the difference in spread is real; the percentiles say where it sits.

Why it matters

A line that fills 500 ml on average but swings by 13 ml wastes product and fails inspections; an insurance book with a fatter tail needs more capital. Both decisions (re-tune the line, price the capital) are about spread, not the mean. Testing spread with Levene means the answer holds whatever the shape of the data.

When this goes wrong

The χ² test for one variance and the F test for two are exact only for normal data. With skewed or heavy-tailed data their false-alarm rates can be several times α, and no sample size fixes that: the animation's skewed line raised them to 13.5 %. Levene's and the Fligner–Killeen tests are robust to that but less powerful when the data really are normal, so they can miss a moderate difference in small samples, as they did on the two lines. None of them rescues dependent data: bottles from one batch, or claims from one storm, share their spread.

In code

chi2, f, levene and fligner are in scipy.stats. Excel's F.TEST returns the two-sided p-value directly. Excel has no Levene or Fligner–Killeen test: W is a one-way ANOVA on the absolute deviations from each line's median (the Data Analysis ToolPak's "Anova: Single Factor"), then F.DIST.RT; the Fligner–Killeen route is not practical in a spreadsheet.