Paired proportions: McNemar's test
H₀: the two measures are positive equally often · H₁: the two measures differ
χ² = 39.0508 · df = 1 · p = 4.13E-10
p = 4.13E-10 ≤ α = 0.05: reject H₀. The disagreements split 5 against 54, more unevenly than a fair coin would easily produce: the two measures differ.
How to read the picture
For the χ² forms, the curve is the null world of χ² with one degree of freedom; for the exact form, the bars are the binomial distribution of c among the b + c disagreements at ½.
The vertical line is your result; the shaded area or the highlighted bars are the p-value.
Below, the odds ratio c / b with its exact interval: above 1, the second measure wins more of the disagreements.
Your data
Method
Significance level α
When two models, two raters or two moments are scored on the same cases, the right comparison is paired. Cases where both are right, or both wrong, say nothing about which is better; only the disagreements count. McNemar's test asks whether they split evenly.
The estimate and what the data show
The worked steps
The four boxes: [[both, only the first], [only the second, neither]]
[[954, 5], [54, 87]] · N = 1100The two measures' rates
p̂₁ = (954 + 5) / 1100 = 0.8718 · p̂₂ = (954 + 54) / 1100 = 0.9164The disagreements, and the second measure's share of them
b + c = 5 + 54 = 59 · c / (b + c) = 0.9153The χ² statistic
χ² = (|b − c| − 1)² / (b + c) = (|5 − 54| − 1)² / 59 = 39.0508The p-value
p = P(χ²(1) ≥ 39.0508) = 4.13E-10Critical value at α
α = 0.05: χ²* = 3.8415The odds ratio of the disagreements
OR = c / b = 54 / 5 = 10.8Exact interval for the odds ratio
95 %: [4.3535, 34.5983]
The same in code
- statsmodels
from statsmodels.stats.contingency_tables import mcnemar mcnemar([[954, 5], [54, 87]], exact=False, correction=True)- scipy: the exact form
from scipy import stats stats.binomtest(54, 59, 0.5)- Excel
=CHISQ.DIST.RT((ABS(5-54)-1)^2/59, 1)
Formulas
If the two measures are equally good, each disagreement is equally likely to go either way, like a fair coin: b and c should be close.
Try this
Only the disagreements count
Open Meridian H4 and change 'both right' from 954 to 9,540.
Look for: Nothing moves: the test statistic, the p-value and the interval depend on b and c alone. Agreement is not evidence about which model is better.
χ² against exact
Open the close call and switch between the corrected χ², the plain χ² and the exact test.
Look for: The continuity correction makes the χ² test closer to the exact one; without it the p-value is smaller. With few disagreements, use the exact test.
The wrong test
Open Meridian H4 and compare the two models' rates: 959 and 1,008 right out of 1,100, a gap of under 5 points.
Look for: Treated as two independent groups, a gap that small in 1,100 cases would look modest; paired, the 5 against 54 disagreements make it overwhelming. The same cases scored twice call for McNemar's test.