Counts over exposure, the same cases scored twice, and a difference small enough to ignore.
The frequency–severity split: loss cost = how often × how much
A diagram of three boxes joined by × and =, drawn for each book. For the standard book: 0.1104 claims per vehicle-year times an average claim of £5,964 equals a loss cost of £658 per vehicle-year. For the telematics pilot: 0.0871 times £4,730 equals £412. Under each box, the change from standard to telematics: frequency −21.1 %, average claim −20.7 %, loss cost −37.4 %.
What you just saw
The picture is the identity that organises this whole review. The cost of claims per vehicle-year, the loss cost, splits into how often policies claim, the frequency, times what a claim costs once it happens, the severity. For Meridian's standard book that is 0.1104 × £5,964 = £658 a vehicle-year; for the telematics pilot, 0.0871 × £4,730 = £412.
Splitting loss cost this way is not presentational: the two parts respond to different levers. Telematics is a frequency intervention. It changes driving behaviour, so it should reduce how often people crash, with little reason to change what a crash costs once it happens. The review's first two hypotheses test exactly that expectation, and a result where both moved would be a reason for suspicion rather than celebration. Keep an eye on the second box: the average claim is lower too.
Meridian Assurance, a fictional UK motor insurer, piloted a telematics-rated product across half its renewal book in accident year 2025: 60,000 policies, 4,663 settled claims. The Pricing Committee must decide three things: whether to roll telematics out across the whole portfolio, whether the tariff's large-loss loading is still adequate, and whether to replace the fraud-detection model. Under those decisions sit seven statistical questions, and three of them need tests the rest of the lab has only named:
Seven questions, seven tests: insurance data break the usual toolkit's assumptions in three ways
#
Question
Data
Test
H1
Does telematics cut claim frequency?
Counts over unequal exposure
Poisson rate-ratio test
H2
Does telematics cut claim severity?
Right-skewed cost, two groups
Welch's t test on log severity
H3
Does it cut severity volatility?
Spread, two groups
Levene's test, with Fligner–Killeen
H4
Do the two fraud models differ?
Paired yes or no, the same claims
McNemar's test
H5
Does severity differ by region?
Skewed cost, four groups
Kruskal–Wallis, then Holm
H6
Is the large-loss rate the tariff's 2.5 %?
One proportion against a constant
Exact binomial test
H7
Is telematics severity equivalent to standard?
Two groups, the burden of proof reversed
TOST, two one-sided tests
Claims are counts observed over unequal exposure. A policy in force for four months cannot be compared with one in force for twelve. A two-proportion test would silently ignore this; the rate test takes the exposure into account. This is why actuaries model claims per vehicle-year, never "the share of policies that claimed".
Severity is lognormal, not normal. Motor claims run from a £200 windscreen to a bodily-injury settlement in the millions. The arithmetic mean is dominated by a handful of losses, so the review tests on the log scale and reports a ratio of geometric means.
Risk is spread, not just the mean. A book with the same expected cost but fatter tails needs more capital. H3 promotes what is usually an assumption check to a hypothesis in its own right.
The datasets
The data are the notebook's own, exported row for row. The policies file has 60,000 rows and is a larger download; the chapter reads a per-tier summary of it that a test rebuilds from the full file. The README says what every column holds.
meridian-policies.csv· 60,000 rowsOne row per policy-year in accident year 2025: the frequency grain, with the exposure each policy earned.
meridian-claims.csv· 4,663 rowsOne row per settled claim: the severity grain, which exists only for the policies that claimed.
meridian-fraud-review.csv· 1,100 rowsOne row per claim referred to the investigations unit, where the true fraud status is known, scored by both fraud models.
The review's thirteen stages as a row of numbered steps: step 1, two grains and five gates, is marked as the current one, and the rest are still to come.
The review reads two tables at two different grains, the defining structural feature of insurance data. The policies table has one row per policy-year, carrying the exposure (earned vehicle-years) and the claim count: the frequency grain. The claims table has one row per settled claim, carrying the amount: the severity grain, which exists only for the minority of policies that claimed.
Confusing the two grains is the commonest error in insurance analysis: an average severity worked out over policies instead of over claims divides by the wrong number and understates severity by an order of magnitude.
The five gates on Meridian's data: all pass
Gate
What it checks
On Meridian's data
1. Grain
One row per policy
0 duplicated policy ids in 60,000 rows
2. Exposure
Earned exposure in (0, 1.05] vehicle-years
0 policies outside it
3. Referential integrity
Every claim belongs to a known policy
0 orphan claims
4. Amounts
Claim amounts are positive
0 non-positive amounts
5. Reconciliation
Claim rows equal the policies' claim counts
4,663 rows, 4,663 counted
The exposure gate is the insurance-specific one. Exposure above one vehicle-year on an annual policy usually means rows duplicated by a mid-term adjustment, and because exposure sits in the denominator of every frequency, it deflates the claim rate across the entire book without producing a single obviously wrong number.
Why the exposure gate matters
A mid-term adjustment duplicates some policy rows, so a few policies show 1.4 vehicle-years on an annual policy. No single number looks wrong. What does it do to the review?
The Excel lines assume the two files open as sheets named policies and claims, with their columns in order.
Step 2: cohort balance, where a large p-value is the good news
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: step 1 is built, step 2, the cohort balance, is marked as the current one, and the rest are still to come.
Before asking whether telematics worked, ask whether the two cohorts were comparable to begin with. If the pilot arm had drawn shorter policies or a different regional mix, every result after this would be confounded. Two checks, with the logic inverted: here we want large p-values, and the review uses α = 0.01 for them.
The cohort-balance checks: both pass at α = 0.01
Check
Test
Result
Reading
Exposure
Mann–Whitney U (exposure is bounded and skewed, so not a t test)
means 0.786 and 0.788 vehicle-years, p = 0.116
No sign of a difference
Region mix
χ² test of independence, tier × region
χ² = 1.569, df = 3, p = 0.666
No sign of a difference
A failure here would not mean telematics is bad. It would mean the comparison is invalid, and the analysis must stop until the assignment is understood.
Reading the balance check
The cohort-balance check on exposure gives p = 0.116 and on the region mix p = 0.666, both at α = 0.01. What do they tell the review?
Excel has no Mann–Whitney test: B1 holds U turned into a z (from RANK.AVG and the formula). B2:E3 hold the tier × region counts and B6:E7 the expected ones.
Step 3: frequency times severity
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 2 are built, step 3, the frequency–severity split, is marked as the current one, and the rest are still to come.
Symbol by symbol, for the standard book: 2,601 claims over 23,561 vehicle-years is a frequency of 0.1104; its claims cost £15,512,393 in total, an average of £5,964 each; so the loss cost is \(0.1104 \times 5{,}964 = 658\) pounds per vehicle-year. The claims cancel, which is why the product is simply the incurred cost per vehicle-year.
The portfolio by tier, accident year 2025
Standard
Telematics
Policies
29,892
30,108
Exposure (vehicle-years)
23,561
23,672
Claims
2,601
2,062
Frequency (claims per vehicle-year)
0.1104
0.0871
Average claim (arithmetic mean)
£5,964
£4,730
Median claim
£2,149
£2,101
Earned premium
£19.0m
£17.6m
Incurred claims
£15.5m
£9.8m
Loss ratio
81.6 %
55.4 %
Frequency is 21.1 % lower in the telematics book, as the mechanism predicts. But the average claim is also 20.7 % lower, while the median claim barely moved (£2,101 against £2,149). By the review's own rule, both moving is a reason to look harder. Steps 6, 7 and 11 find where the second effect lives: not in the typical claim, but in the large ones.
Loss cost from frequency and severity
The telematics book had 0.0871 claims per vehicle-year and an average claim of £4,730. What is its loss cost per vehicle-year, in pounds? Whole pounds (within 1).
A decimal or a fraction; within 1 of the answer counts as right.
The review's thirteen stages as a row of numbered steps: steps 1 to 3 are built, step 4, the pre-registration, is marked as the current one, and the rest are still to come.
The hypotheses are written and committed before the outcome data are inspected, and stored as data so the audit trail can be read by a machine. For a regulated insurer this is more than good practice: a pricing decision that changes what customers pay must survive review by the board's risk committee and, potentially, the regulator. "We tested this hypothesis, in this direction, decided in advance" is the difference between evidence and rationalisation.
The pre-registration: H1 to H6 form the family; H7 carries its own α
Id
Family
H₀
Hₐ
Tail
Test
H1
Primary
rate_tel = rate_std
rate_tel ≠ rate_std
Two-sided
Poisson rate ratio
H2
Primary
mean log severity equal
they differ
Two-sided
Welch's t on logs
H3
Primary
variances of log severity equal
they differ
Two-sided
Levene
H4
Secondary
discordant pairs symmetric
the models differ
Two-sided
McNemar
H5
Secondary
the four regional distributions identical
at least one differs
Right (H)
Kruskal–Wallis and Holm
H6
Primary
P(loss > £25,000) = 0.025
it differs
Two-sided
Exact binomial
H7
Confirmatory, outside the family
|difference| ≥ margin
|difference| < margin
Two one-sided
TOST
Two design choices are worth defending. H1 and H2 are two-sided even though a reduction is hoped for: a one-sided test would have more power in the hoped-for direction, but it could never report harm, and a telematics product that attracts a worse risk pool is exactly what the pilot exists to catch.
And H7 reverses the burden of proof. The null of an equivalence test is that a material difference exists; rejecting it is what lets you claim equivalence. Its error structure runs the other way from the superiority tests, so it carries its own α and stays outside the family (step 12).
Spot the mistake: H4's tails
Spot the mistake
The notebook's pre-registration lists H4 as two-sided while its alternative reads "the challenger is more often correct". What is the consistent fix?
multipletests is in statsmodels.stats.multitest; the Excel line is one row of BH, with the running minimum statsmodels also applies.
Step 5: H1, claim frequency, a rate over exposure
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 4 are built, step 5, H1, claim frequency, is marked as the current one, and the rest are still to come.
Exposure matters: Meridian's year replayed week by week
Playback speed
Two panels fill in over the 52 weeks of accident year 2025. The first shows the share of each book's policies that have claimed so far: it keeps climbing as time on risk accumulates, so it depends on how long the policies were in force. The second shows claims per vehicle-year for each book: after a noisy start it settles, because it divides by the exposure actually earned. Below them, the rate ratio of telematics to standard and its 95 % interval narrow week by week and land on the year's result, 0.789 with an interval from 0.745 to 0.836, entirely below the committee's 0.9. The files hold no dates, so the replay places each policy's claims at random moments while it was on risk; the totals at week 52 are the real ones.
Claims are counts observed over unequal exposure, so the comparison is of rates, not proportions. Under a Poisson model, the claims on a policy with exposure \(t_i\) have mean \(\lambda t_i\), where \(\lambda\) is the claim rate per vehicle-year. Summed over a book, the total claims \(Y\) have mean \(\lambda T\), with \(T\) the book's total exposure. The hypothesis is about the rate ratio:
With the numbers: telematics had \(Y_{\text{tel}} = 2{,}062\) claims over \(T_{\text{tel}} = 23{,}672\) vehicle-years, a rate of 0.0871; standard had 2,601 over 23,561, a rate of 0.1104. So \(\widehat{RR} = 0.0871 / 0.1104 = 0.789\): a 21.1 % lower claim frequency. The score test gives z = −8.05 and p = 8.0 × 10⁻¹⁶, and the 95 % score interval for the ratio runs from 0.745 to 0.836.
H1: the rate ratio with three intervals, against no change and the committee's threshold
Three intervals for the rate ratio of telematics to standard, 0.789: the score interval (0.745 to 0.836), the exact conditional one (0.744 to 0.836) and the Wald interval on the log ratio (0.745 to 0.836). They almost coincide. A line at 1 marks no change and a dashed line at 0.9 the committee's 10 % threshold: every interval lies entirely below both.
Why not a two-proportion z test? It would test "what share of policies had a claim", throwing away both the differences in exposure and the policies that claimed more than once. On a book where exposure varies from policy to policy, that is not a rounding error: it is a bias against whichever book happens to have shorter policies. The animation shows it: the share of policies that have claimed is still climbing at the end of the year, while the rate per vehicle-year settled long before.
The effect size is the rate ratio itself. Unlike Cohen's d it needs no table of benchmarks: a pricing actuary reads 0.789 directly as a 21.1 % frequency reduction, which is exactly the input a tariff rebuild needs. And materiality: the committee's threshold is a 10 % reduction, and the interval's upper end, 0.836, still means at least a 16.4 % reduction. The benefit clears the bar even at its least favourable.
Why this is true:The exact conditional test
Given the total number of claims, 4,663, each one is a telematics claim with probability π₀ = T_tel / (T_tel + T_std) = 0.5012 if the rates are equal: the books' shares of the exposure. So the telematics count is binomial, Binomial(4,663, 0.5012), under H₀, and the exact binomial test of 2,062 gives p = 8.4 × 10⁻¹⁶.
The Clopper–Pearson interval for π maps back to the rate ratio as RR = π / (1 − π) × T_std / T_tel. With thousands of claims the exact and score answers agree; with a handful, only the exact one can be trusted.
The rate ratio by hand
Telematics had 2,062 claims over 23,672 vehicle-years; standard had 2,601 over 23,561. What is the rate ratio, telematics ÷ standard? Three decimals.
A decimal or a fraction; within 0.001 of the answer counts as right.
Why not the share of policies that claimed?
Why does the review compare claims per vehicle-year rather than the share of policies that had a claim?
test_poisson_2indep and confint_poisson_2indep are in statsmodels.stats.rates, binomtest in scipy.stats. Excel has no rate test: the first line is the score z by formula (turn it into p with NORM.S.DIST), the second the Wald interval on the log ratio with the ratio in B1, which is close to the score interval at these counts, and the third the exact route, doubling the smaller tail.
Step 6: H2, claim severity, on the log scale
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 5 are built, step 6, H2, claim severity, is marked as the current one, and the rest are still to come.
H2: two ratios of severity, the geometric mean and the arithmetic mean
Two intervals on a scale of ratios, telematics to standard. The geometric mean ratio, 0.930, has a 95 % interval from 0.860 to 1.006, which just includes 1: no significant change in the typical claim. The arithmetic mean ratio, 0.793, has a 95 % interval from 0.711 to 0.885, well below 1: the average claim fell. They answer different questions.
Claim amounts are heavily right-skewed: the raw skewness is 6.14, and on the log scale it is 0.05, close to symmetric. So H2 tests the difference in mean log severity, which is the same as testing the ratio of geometric means:
Exponentiating the difference and its interval gives a factor that reads directly as "telematics claims cost X % less". The geometric means are £2,067 and £2,223; the difference in logs is \(-0.0726\) with a standard error of 0.0399, so Welch's t = −1.82 on about 4,555 degrees of freedom and p = 0.069. The ratio is \(e^{-0.0726} = 0.930\), 95 % interval 0.860 to 1.006: not significant at 5 %.
Why the log scale rather than a t test on raw amounts? At this size a raw-scale Welch test is not invalid: the sampling distribution of a mean does become normal eventually. But it answers a question nobody asked. The arithmetic mean of a lognormal is dominated by a few extreme losses, so the raw test mostly detects whether one book happened to draw a large bodily-injury claim. On the log scale the estimate is stable and the effect comes out as a factor, the way pricing uses it. And Welch rather than Student, because there is positive reason to expect unequal variances: that is H3.
Three tests, three questions
Test
The question it answers
Statistic
p
Welch's t on log amounts
Did the typical claim (the geometric mean) change?
t = −1.82
0.069
Mann–Whitney U
Does one book tend to produce larger claims?
U = 2,615,526
0.148
Welch's t on raw amounts
Did the average claim (the arithmetic mean) change?
t = −4.10
4.3 × 10⁻⁵
The log-scale and rank-based tests agree, and the raw-scale test disagrees with both. That is not a contradiction to explain away: it is three tests answering three different questions. For a lognormal, the mean is exp(μ + σ²/2), so the arithmetic mean absorbs the difference in spread that H3 tests directly: the mean is 2.25 times the median in the telematics book and 2.78 times in the standard one.
Which mean does the decision need?The typical claim and the average claim are different estimands. The geometric mean describes what a typical claim costs; the arithmetic mean is the expected cost of a claim, which is what a tariff prices. Keep both in view: step 11 shows why it matters.
Three tests, three questions
On the same claims, Welch on log amounts gives p = 0.069, Mann–Whitney p = 0.148 and Welch on the raw amounts p = 4.3 × 10⁻⁵. Which reading is right?
Columns B and C hold the telematics and standard claim amounts. In Excel, T.TEST with LN(...) must be entered as an array formula in older versions; B1 holds Mann–Whitney's U as a z.
Step 7: H3, severity volatility, a spread worth money
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 6 are built, step 7, H3, volatility, is marked as the current one, and the rest are still to come.
H3: telematics severity as a share of standard, from the median to the far tail
Six points on a scale of ratios, one per quantile of the claim amounts: the telematics book's quantile divided by the standard book's. At the median the ratio is 0.98; it falls steadily into the tail, to 0.70 at the 99th percentile. A line at 1 marks no difference. The typical claims are alike; the large ones are not.
This is the test usually demoted to an assumption check, promoted here to a hypothesis. An insurer cares about spread directly: capital requirements under Solvency II, the European rules that set how much capital an insurer must hold, are driven by the tail of the loss distribution, not its mean. Two books with the same expected cost but different spread do not consume the same capital, and do not attract the same reinsurance price.
Why Levene rather than the F test of two variances? The F test is notoriously sensitive to departures from normality, and severity, even on the log scale, has heavier tails than a normal. Levene's test compares the absolute distances of each claim from its group's centre; centred on the median (sometimes called the Brown–Forsythe version) it stays valid under exactly this kind of skew. Fligner–Killeen, a rank-based alternative, gives a second opinion.
Spread and the tail, by tier
Measure
Telematics
Standard
Standard deviation of log severity
1.3038
1.4122
Coefficient of variation, raw amounts
1.771
2.036
50th percentile
£2,101
£2,149
75th percentile
£4,972
£5,910
90th percentile
£11,272
£14,087
95th percentile
£17,190
£23,502
99th percentile
£43,655
£62,423
99.5th percentile
£57,107
£74,155
The variance ratio, standard to telematics on the log scale, is 1.17. Levene's test gives W = 18.29 on (1, 4,661) degrees of freedom, p = 1.9 × 10⁻⁵; Fligner–Killeen gives p = 1.8 × 10⁻⁵. Both are below 0.05, so here they agree, and the conclusion is not an artefact of the method. The telematics book is less volatile, and at the 99th percentile its claims are 30 % smaller.
The variance ratio
On the log scale, standard claims have a standard deviation of 1.4122 and telematics claims 1.3038. What is the variance ratio, standard ÷ telematics? Three decimals.
A decimal or a fraction; within 0.001 of the answer counts as right.
What it computes
Python (scipy, statsmodels)
Excel
Levene's test on log severity, centred on the median
levene(np.log(tel), np.log(std), center='median')
=F.DIST.RT(18.29, 1, 4661)
Fligner–Killeen, the second opinion
fligner(np.log(tel), np.log(std))
=CHISQ.DIST.RT(18.43, 1)
A tail quantile
np.quantile(tel, 0.99)
=PERCENTILE.INC(B2:B2063, 0.99)
levene and fligner are in scipy.stats. Excel has neither: the lines turn each statistic into its p-value. PERCENTILE.INC interpolates exactly as numpy's default does.
Step 8: H4, two fraud models on the same claims
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 7 are built, step 8, H4, the fraud models, is marked as the current one, and the rest are still to come.
Only the disagreements count: 1,100 claims sorted by what both fraud models said
Playback speed
The 1,100 investigated claims drop, 50 at a time, into the four boxes of a 2 × 2 table: both models right (954), only the champion right (5), only the challenger right (54), both wrong (87). Then the 1,041 claims on which the models agree fade out: they say nothing about which model is better. The 59 disagreements remain, and the question becomes a coin toss: if the models were equally good, each disagreement would go either way with probability one half. The last frame draws the binomial distribution of 59 tosses and marks 5, so far in its tail that the exact two-sided p-value is 1.9 × 10⁻¹¹.
The 1,100 claims referred to the investigations unit, where the true fraud status is known, were each scored by the incumbent model (the champion) and the new one (the challenger): paired yes-or-no outcomes. The champion was right on 87.2 % of them and the challenger on 91.6 %.
The fraud review as a 2 × 2 table of the same claims
Challenger wrong
Challenger right
Champion wrong
87
54
Champion right
5
954
Why McNemar and not a two-proportion test? The two accuracies come from the same claims, so they are not independent samples, and a two-proportion test would get the standard error wrong. McNemar sets aside the claims both models got right and both got wrong, which say nothing about which is better, and tests only the discordant pairs:
Here \(b = 5\) claims had only the champion right and \(c = 54\) only the challenger; the \(-1\) is a continuity correction for counting in whole claims. So \(\chi^2 = (|5 - 54| - 1)^2 / 59 = 39.05\) on 1 degree of freedom, p = 4.1 × 10⁻¹⁰. Under H₀ a disagreement is equally likely to fall either way, so it is, in essence, a sign test on the disagreements: the exact version, a binomial test of 5 in 59 at one half, gives p = 1.9 × 10⁻¹¹.
The effect size is the odds that a disagreement goes the challenger's way: c/b = 10.8, with an exact 95 % interval from 4.4 to 34.6. Note the denominator: the test rests on the 59 discordant claims, not all 1,100. The wrong, unpaired two-proportion test on the same data gives p = 0.00069: still significant here, but far weaker evidence, because it treats one set of claims as two.
The most misapplied test in model validationComparing two models on the same holdout set is a paired problem. Treating it as two independent samples is both common and wrong: always ask whether the two numbers you are comparing came from the same cases.
McNemar from the discordant pairs
On the same 1,100 claims, the champion alone was right on 5 and the challenger alone on 54. What is McNemar's continuity-corrected χ²? Two decimals.
A decimal or a fraction; within 0.01 of the answer counts as right.
Why the unpaired test is wrong here
Both fraud models scored the same 1,100 claims. Why is a two-proportion z test on their accuracies the wrong test?
mcnemar is in statsmodels.stats.contingency_tables; its table has the champion's wrong and right as rows and the challenger's as columns.
Step 9: H5, severity by region, and which regions
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 8 are built, step 9, H5, the regions, is marked as the current one, and the rest are still to come.
H5: the difference in median claim for each pair of regions
Six points, one per pair of regions, each the difference in median claim amount in pounds, against a line at zero. The three pairs with London sit well to the right: London's median claim, £2,488, is the highest. Those three survive Holm's correction; the other three, drawn fainter, do not.
A skewed outcome in four independent groups. ANOVA on raw severity would test the means of a lognormal with unequal spreads across regions: two broken assumptions at once. Kruskal–Wallis is the rank-based analogue. It makes no assumption about the distribution's shape and asks whether one region tends to produce larger claims than another, which for a regional rating factor is arguably the more relevant question anyway.
The test gives H = 42.6 on 3 degrees of freedom, p = 3.0 × 10⁻⁹: at least one region differs. Its effect size, ε² = (H − k + 1)/(N − k) = 0.0085, is the rank version of η²: the share of the variation in ranks that region accounts for. Then the post-hoc problem: six pairwise comparisons at α = 0.05 would carry about a 26 % chance of at least one false positive, so the pairwise Mann–Whitney p-values are corrected with Holm, which is never less powerful than Bonferroni and controls the same family-wise error rate.
Pairwise Mann–Whitney tests with Holm's correction: three of six pairs differ
Pair
Median difference
p
Holm adjusted
Significant?
London vs Midlands
£391
1.5 × 10⁻⁴
5.8 × 10⁻⁴
Yes
London vs North
£692
6.0 × 10⁻⁹
3.6 × 10⁻⁸
Yes
London vs Scotland
£591
9.1 × 10⁻⁶
4.6 × 10⁻⁵
Yes
Midlands vs North
£301
0.0232
0.0696
No
Midlands vs Scotland
£200
0.1248
0.2496
No
North vs Scotland
−£100
0.7612
0.7612
No
3 of the 6 pairs survive: London against each of the others. The other regions cannot be told apart.
Spot the mistake: "a few percent"
Spot the mistake
Kruskal–Wallis on severity by region gives H = 42.6 with k = 4 regions and N = 4,663 claims. The notebook's limitations call the effect "a few percent". What is ε² = (H − k + 1)/(N − k)? Four decimals.
A decimal or a fraction; within 0.0001 of the answer counts as right.
What it computes
Python (scipy, statsmodels)
Excel
Kruskal–Wallis across the four regions
kruskal(london, midlands, north, scotland)
=CHISQ.DIST.RT(42.6, 3)
ε²
(h - 4 + 1) / (len(claims) - 4)
=(42.6-4+1)/(4663-4)
The pairs, Holm-corrected
multipletests([mannwhitneyu(a, b).pvalue for a, b in pairs], method='holm')
=MIN(1, (6-C2+1)*B2)
kruskal and mannwhitneyu are in scipy.stats. Excel has no Kruskal–Wallis test: with H worked out from RANK.AVG columns, its p-value is the χ² tail on k − 1 degrees of freedom.
Step 10: H6, the large-loss rate against the tariff
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 9 are built, step 10, H6, the large losses, is marked as the current one, and the rest are still to come.
H6: the large-loss rate with its exact interval, against the tariff's 2.5 %
On a scale of percentages, the share of claims above £25,000: 3.67 % for the whole book with an exact 95 % interval from 3.15 % to 4.25 %, entirely to the right of the tariff's 2.5 % line. Below it, fainter and labelled exploratory, the same rate by tier: 4.46 % for standard and 2.67 % for telematics.
This is the test with the most direct financial consequence. The tariff was priced assuming that 2.5 % of claims breach the £25,000 large-loss threshold, and that one number feeds the large-loss loading in every premium the company charges. If experience says otherwise, the tariff is mispriced. It is one observed proportion tested against a fixed constant, not against another group:
171 of the 4,663 claims exceed £25,000: 3.67 %. The exact binomial test gives p = 1.6 × 10⁻⁶, and the Clopper–Pearson 95 % interval runs from 3.15 % to 4.25 %: the whole interval sits above the tariff's assumption. Experience exceeds the priced rate by 1.17 percentage points, 46.7 % more in relative terms, and large losses account for 32.4 % of all incurred cost.
Why exact rather than the normal approximation? With a small underlying probability the normal approximation is poor in exactly the tail that matters. The exact test works out the probability from the binomial distribution itself, and the Clopper–Pearson interval is guaranteed to reach its stated coverage rather than approximate it. With money attached to the answer, the exact method is the defensible one.
A material finding raises a follow-up: is the breach in one book? The rates by tier, 116 of 2,601 standard claims against 55 of 2,062 telematics claims, differ (χ² with Yates's correction, p = 0.0016). But this comparison was not pre-registered, so it is exploratory: it generates a hypothesis for the next review, not a decision in this one. Labelling it so is the discipline that keeps the pre-registration meaningful.
The large-loss rate
171 of the 4,663 claims exceed £25,000. The tariff assumes 2.5 %. What is the observed large-loss rate, as a percentage to two decimals?
A decimal or a fraction; within 0.01 of the answer counts as right.
binomtest and chi2_contingency are in scipy.stats. The Excel p-value doubles the upper tail, a close stand-in here for scipy's exact two-sided rule; BETA.INV gives the interval's lower end (the upper end is BETA.INV(0.975, 172, 4663-171)).
Step 11: H7, proving no material difference
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 10 are built, step 11, H7, equivalence, is marked as the current one, and the rest are still to come.
Two one-sided tests: the 90 % interval against the ±22 % band, for the geometric and then the arithmetic mean
Playback speed
A band from 0.819 to 1.221 marks the Pricing Committee's margin: a severity ratio inside it is no material change. First the geometric mean ratio, 0.930, grows its 90 % interval step by step to 0.871 to 0.993. Its lower end clears 0.819, so the first one-sided test rejects "at least 18 % cheaper" (p = 0.0007); its upper end clears 1.221, so the second rejects "at least 22 % dearer": equivalence is shown for the geometric mean. Then the arithmetic mean ratio, 0.793, the one the tariff prices, grows its 90 % interval to 0.724 to 0.869. Its lower end lies below 0.819: it is not equivalent, and "the tariff can treat severity as unchanged" does not follow (correction C29).
H2 failed to reject. The universal temptation is to write "telematics has no effect on severity", and that is not what a non-significant result means. Absence of evidence is not evidence of absence: the same p-value comes from a genuinely zero effect and from a study too small to see a real one.
An equivalence test fixes this by reversing the burden of proof. Define an equivalence margin Δ, the largest difference the business would treat as unimportant, and test two nulls, one on each side:
Reject both and there is positive evidence that the true difference lies inside ±Δ; the pair's p-value is the larger of the two. The margin is ±0.20 on the log scale, which is ×0.819 to ×1.221 on the geometric mean, set by the Pricing Committee as the point at which a severity change would alter the tariff. The lower test gives t = 3.19, p = 0.0007; the upper test t = −6.83, p = 4.7 × 10⁻¹². The TOST p-value is 0.0007, and the 90 % interval for the difference in logs, −0.138 to −0.007, lies inside ±0.20. Equivalence is shown, for the geometric mean.
The margin must be a business decision, made in advance. Choosing Δ after seeing the data, wide enough to guarantee "equivalence", is the equivalence-testing version of p-hacking. And the test uses a 90 % interval, not 95 %: each one-sided test runs at 5 %, and together they match an interval with 5 % in each tail.
Reading H2 and H7 together, corrected
Result
What it licenses you to say
H2
Fail to reject
No significant change in the typical claim was detected
H7
Equivalence shown on the geometric mean
And the typical claim changed by less than the committee's ±22 %
The arithmetic mean
Not equivalent
But the average claim, which the tariff prices, is lower: ratio 0.793, 90 % interval 0.724 to 0.869
Only together do they support a decision-grade conclusion: telematics works mainly on frequency, the typical claim is unchanged within the committee's band, and the average claim is lower too, because the telematics book has fewer large losses. The tariff should price that, not assume it away.
Spot the mistake: "the tariff can treat severity as unchanged"
Spot the mistake
TOST on log amounts shows the geometric mean ratio is within ±22 % (90 % interval 0.871 to 0.993). The notebook concludes: "the tariff can treat severity as unchanged". Why does that not follow?
ttost_ind is in statsmodels.stats.weightstats. In Excel, B1 holds the difference in mean logs, B2 its standard error and B3 Welch's degrees of freedom; the TOST p-value is the larger of the two one-sided ones. B4 to B10 hold the ratio, then each book's standard deviation, count and mean.
Step 12: the family, and what H2's null result means
Where this step sits in the review
The review's thirteen stages as a row of numbered steps: steps 1 to 11 are built, step 12, the family and power, is marked as the current one, and the rest are still to come.
The family: six p-values against the corrections
Six horizontal bars, one per hypothesis, each as long as −log₁₀ of its p-value, capped at 10. A line marks α = 0.05 and another Bonferroni's α/6; a tick on each bar marks its Benjamini–Hochberg threshold. H1, H3, H4, H5 and H6 reach past every line; H2, at p = 0.069, reaches none.
Six pre-registered tests at α = 0.05 carry about a 26 % chance of at least one false positive if every null were true and the tests independent. Benjamini–Hochberg, pre-registered, controls the false discovery rate: the expected share of the rejections that are false, better matched to a review where a few false leads are tolerable but a missed deterioration is not. Bonferroni, reported beside it, controls the family-wise error rate.
The family: Benjamini–Hochberg, Bonferroni and Holm on H1 to H6
Hypothesis
p
BH adjusted
Bonferroni adjusted
Holm adjusted
Decision (BH)
H1
8.0 × 10⁻¹⁶
4.8 × 10⁻¹⁵
4.8 × 10⁻¹⁵
4.8 × 10⁻¹⁵
Reject H₀
H2
0.0689
0.0689
0.4132
0.0689
Fail to reject H₀
H3
1.9 × 10⁻⁵
2.3 × 10⁻⁵
1.2 × 10⁻⁴
3.9 × 10⁻⁵
Reject H₀
H4
4.1 × 10⁻¹⁰
1.2 × 10⁻⁹
2.5 × 10⁻⁹
2.1 × 10⁻⁹
Reject H₀
H5
3.0 × 10⁻⁹
6.0 × 10⁻⁹
1.8 × 10⁻⁸
1.2 × 10⁻⁸
Reject H₀
H6
1.6 × 10⁻⁶
2.4 × 10⁻⁶
9.5 × 10⁻⁶
4.8 × 10⁻⁶
Reject H₀
Benjamini–Hochberg, Bonferroni and Holm agree on every hypothesis: H1, H3, H4, H5, H6 are rejected and H2 is not, so the conclusions do not depend on the method. H7 is excluded from the family, deliberately: its null is the presence of an effect, so its error structure runs the opposite way, and pooling it with the superiority tests would be a category error. It carries its own α and is reported on its own.
Power to detect a 10 % frequency reduction, by vehicle-years per arm
A curve rising with exposure per arm, from low power at a few thousand vehicle-years towards 1. A line marks 80 %. It is crossed at 13,522 vehicle-years per arm, marked; the pilot's 23,561 vehicle-years in the standard book sit well past it.
A non-significant result is only interpretable beside the effect the study could have detected. Two calculations. First, the exposure needed to detect the committee's 10 % frequency reduction, in vehicle-years, the unit that drives a Poisson test's precision, not policies:
With \(\lambda_1 = 0.1104\), \(\lambda_2 = 0.9 \times \lambda_1 = 0.0994\) and \(\ln 0.9 = -0.1054\), it comes to 13521.2, so 13,522 vehicle-years per arm, rounded up. The pilot had about 23,561 in each: adequately powered.
Second, the smallest change in severity the study could reliably detect, given the claims it had: (z₀.₉₇₅ + z₀.₈₀) × the standard error of the log difference, 0.0399, gives 0.1118 on the log scale, a 11.8 % change in the geometric mean. The observed change, 7.0 %, sits below that, which is why H2 is not significant; H7 then bounds the true change inside the committee's band.
Exposure for a 10 % reduction
To detect a 10 % frequency reduction from 0.1104 claims per vehicle-year with 80 % power at α = 0.05, how many vehicle-years per arm are needed? A whole number, rounded up.
In Excel, B1 holds the standard book's claim frequency and B2 the log difference's standard error. statsmodels' power_poisson_ratio_2indep in statsmodels.stats.rates gives the same exposure.
Why it matters
Three decisions ride on this review, each on a different test, and each would have gone wrong with the usual toolkit: a share of policies that claimed would have mixed exposure into the frequency, an unpaired test would have understated the fraud model's win, and a non-significant severity test read as "no effect" would have left the tariff overpriced on large losses and blind to the cheaper average claim. And the money: underwritten at the point estimate, the rollout is a £60.6m story; at the interval's bound it is £47.1m. The second is the number the business can defend.
When this goes wrong: the review's limitations
Every one of these would be raised in a competent review, so they belong in the readout, not in the reviewer's notes.
Claims development. Every figure is based on claims reported and settled within the accident year. Bodily-injury claims can take years to settle, and the largest take longest. Severity and the large-loss rate are both understated, in the direction that makes H6 worse, not better. A production version would develop the claims to their ultimate cost, or use a fully developed accident year.
Self-selection is assumed away. The pilot is treated as randomised at postcode-cohort level, and the balance checks pass. In the real world customers choose telematics, and the drivers who opt in are more careful, so a naive comparison credits the product with a selection effect. If assignment is not genuinely random, the honest options are propensity-score matching or an instrumental-variable design, not this analysis.
One accident year, one country. No allowance for weather, claims inflation or the rising cost of repairing electric vehicles. A single year cannot separate a telematics effect from a benign-weather effect.
Independence within policies. The Poisson model treats claims as independent, which the extra spread in real motor data routinely violates: a small group of drivers makes many claims. A negative binomial model would be the standard remedy, and would widen H1's interval.
Statistical significance is not practical significance. H5 is highly significant on 4,663 claims, but ε² = 0.0085: region explains under 1 % of the rank variation (correction C35). Significance answers "is it real"; effect size answers "does it matter". Both belong in the readout; only the second belongs in the business case.
And the data are the notebook's synthetic book. The method transfers to a real review; the numbers do not.
The notebook's order, which is the transferable skeleton: configuration, gates, balance, pre-registration, one section per hypothesis, the family, power, results as data, the memo, the limitations. The tests change with the domain; the skeleton does not.
Try thisCompare the two tiers' claim frequencies in the rates explorer, with the score, Wald and exact tests.
The review's thirteen stages as a row of numbered steps: steps 1 to 12 are built, step 13, the memo and the limits, is marked as the current one, and the rest are still to come.
The last thing the review produces is the only part most of its audience will read. It contains no p-values: the Pricing Committee needs the decisions, the size of each effect and the uncertainty around it. The statistics are the evidence, not the message. One check first: the notebook's own summary told the committee to underwrite at the lower bound, and then printed only the point estimate.
Spot the mistake: underwriting at the point
Spot the mistake
The notebook's summary says "underwrite at the lower bound" but prints the saving at the point estimate, £60.6m. The rate ratio's interval runs up to 0.8360. What is the gross saving at that bound, in £ millions? One decimal.
A decimal or a fraction; within 0.1 of the answer counts as right.
B1 holds the standard book's claim frequency, B2 the rate ratio's upper 95 % bound, B3 the average claim and B4 the rate ratio.
Decision memo
Portfolio risk review: RSK-2026-Q3-TELEMATICS
Meridian Assurance · motor book, accident year 2025 · for the Pricing Committee
Recommendation
1.
Proceed with the portfolio-wide telematics rollout, with the business case underwritten at the lower bound of the frequency benefit.
2.
Re-rate the large-loss loading before the next tariff filing.
3.
Promote the challenger fraud model to champion.
4.
Price telematics severity on the average claim, which is lower, rather than treating severity as unchanged.
Telematics: frequency benefit confirmed
Claim frequency
0.1104 → 0.0871 claims per vehicle-year: a 21.1 % reduction, 95 % interval 16.4 % to 25.5 %, above the committee's 10 % even at its lower end.
Severity
The typical claim is unchanged within the committee's ±22 % (geometric mean ratio 0.930). The average claim, which the tariff prices, is lower: ratio 0.793, 90 % interval 0.724 to 0.869.
Volatility
Severity is less dispersed in the telematics book (variance ratio 1.17; its 99th-percentile claim is 30 % smaller), supporting a lower capital charge.
Scaled to the book
Across 480,000 vehicle-years, about 11,178 fewer claims a year at the point estimate, £60.6m gross; at the interval's bound, 8,690 fewer claims, £47.1m gross of the telematics running cost. Underwrite at £47.1m.
Large-loss rate: tariff assumption breached
Observed
3.67 % of claims exceed £25,000, against the tariff's 2.50 %; 95 % interval 3.15 % to 4.25 %, entirely above it.
Consequence
A gap of 1.17 points: a pricing-adequacy issue independent of telematics, to escalate to the next tariff review.
Fraud model: the challenger outperforms
Accuracy
87.2 % → 91.6 % on 1,100 investigated claims. The models disagreed on 59 claims, and the challenger was right on 54 of them.
Portfolio context and caveats
Book
Earned premium £36.6m, incurred claims £25.3m, loss ratio 69.0 %.
Regions
Severity differs by region (London is highest), but region explains under 1 % of the variation: revalidate the regional factors alongside the large-loss loading, without expecting much from them.
Limits
One accident year, claims not yet fully developed, assignment assumed random: the large-loss rate is more likely understated than overstated.
That is the end of the lab. The same skeleton ran two very different reviews: fix the plan before the data, check the data and the comparison before any hypothesis, match each question to the test its data need, give every result its interval and its threshold, correct the family as promised, and write the decision with the money at the interval's bound. Chapter 19 is the same pipeline on an A/B test; chapters 16 and 18 go deeper into estimands, equivalence and the family.
Remember:Match the test to the data and the estimand to the decision. Counts over unequal exposure are compared as rates; the same cases scored twice are compared on their disagreements; "no difference" is proved with an equivalence test against a margin fixed in advance, on the mean the decision actually uses. And the business case is underwritten at the interval's bound, not its point.
Three questions on the chapter. Your best score is kept in this browser.
Answer every question first.
Two readouts: a workshop · step 6 of 6
Equivalence, and the estimand
Two one-sided tests show equivalence for the geometric mean severity. But the tariff prices the arithmetic mean, whose ratio is 0.793: a fifth lower, not unchanged. End on the lesson: the estimand must match the decision, and the memo says so.