Paired samples
Same people, before and after: why is that a different test, and what does pairing buy?
A slope chart draws each call-centre agent as a line from their average daily calls before the rollout to their average after. The lines then collapse into one column of differences, each agent's own change, and a one-sample t test of those differences against 0 draws its 95 % interval, −5.11 to −2.66 calls. Beside it, the wrong analysis, which treats the two columns as independent groups, draws a much wider interval, −6.97 to −0.80. Then the same is played for the first 12 agents only: the paired test still finds the change (p = 0.0067), the independent test does not (p = 0.26).
What you just saw
Helion also measured its call-centre agents twice: each agent's average daily calls in the four weeks before Portal v2 went live and in the four weeks after. Eighty agents, two columns. Each line in the slope chart is one agent, and most of them slope down.
Then the lines collapsed into a single column: each agent's own change, after minus before. That column is all a paired test looks at. Its mean is −3.89 calls a day, and its 95 % interval, −5.11 to −2.66, sits well clear of zero. The second bar is what you get if you forget that the columns belong to the same people and compare them as two independent groups: the same mean change, but an interval from −6.97 to −0.80, more than twice as wide. Same data, different design assumption, very different certainty.
The replay with only the first 12 agents makes the point sharper: the paired test still finds the change (p = 0.0067), while the independent analysis does not (p = 0.26). The verdict flips on the choice of test, and the design, not the p-value, decides which test is right.
Independent or paired?
This is decided by the study design, not by the data, and getting it wrong is a real error rather than a matter of style. Ask: does each observation in group 1 have a natural partner in group 2?
- Customers on the old portal against customers on Portal v2: different people, no partners, so independent.
- Each agent's calls before and after the rollout: the same agent appears twice, so paired.
- Patients matched on age and sex, one in each arm of a trial: deliberately partnered, so paired.
Treating paired data as independent throws the pairing away and badly reduces power, as the animation showed. Treating independent data as paired is worse: it is simply invalid, because there are no partners to subtract.
The paired t test
The paired t test computes the difference for each unit and then runs a one-sample t test on those differences against zero:
In words: \(\bar{d}\) is the mean of the differences, \(s_d\) their standard deviation and \(n\) the number of pairs, not the number of measurements. For the 80 agents the mean difference is −3.885 calls, with \(s_d\) = 5.50, so the standard error is \(5.50/\sqrt{80}\) = 0.615:
On 79 degrees of freedom that gives p = 1.5 × 10⁻⁸, and a 95 % interval for the mean change of −5.11 to −2.66 calls a day. The effect size for pairs is d_z, the mean difference over the standard deviation of the differences: −3.885 / 5.50 = −0.71, a medium-to-large effect.
What pairing buys you
The power advantage is substantial. The agents handle anything from about 31 to 85 calls a day. That spread between agents swamps a 4-call improvement if you compare the two columns as independent samples. Compute each agent's own change and the between-agent variation cancels out entirely, leaving only the variation in the change itself.
| Analysis | t | p-value | Verdict |
|---|---|---|---|
| Paired t test (right) | −6.32 | 1.5 × 10⁻⁸ | Strong evidence |
| Independent t test (wrong) | −2.49 | 0.014 | Weak evidence |
The two p-values differ by a factor of about a million: 5.97 orders of magnitude. The pairing was free, sitting in the data; failing to use it would have buried a real effect, and on the first 12 agents it would have buried it completely.
Why this is true: Why the differences vary so much less than the columns
An agent on a busy queue is busy before and after, so the two columns move together: their correlation is r = 0.85 (chapter 15 tests it). The variance of a difference subtracts twice that shared movement:
For the agents: 89.35 + 105.35 − 164.43 = 30.27, so \(s_d = \sqrt{30.27}\) = 5.50 calls, against standard deviations of 9.45 and 10.26 calls in the two columns. The independent test cannot subtract the shared part, because it does not know the columns are pairs.
What a paired test assumes
The two measurements of one agent are dependent on purpose; that is the point. What must hold is different:
- The pairs are independent of each other: one agent's change tells you nothing about another's.
- The differences are roughly normal, or there are enough pairs for the central limit theorem to work on their mean. It is the differences that matter, not the raw columns.
For the 80 agents the differences are only mildly skewed (skewness 0.46), and a rank-based check that does not assume normality, the Wilcoxon signed-rank test (chapter 14), agrees: p = 6.0 × 10⁻⁸, in the same direction.
The caveat that belongs in the readout
Why it matters
Pairing is often the cheapest power you will ever buy: the same people, measured twice, can show with 12 agents what two separate groups could not show at all. It decides real questions, such as whether the call centre can be staffed for about four fewer calls per agent a day. But it answers "did the calls change?", not "did the portal change them?", and a staffing plan built on the second needs a control group.
When this goes wrong
A paired test is only as good as its pairs. Pairs that are not really partners (two different agents matched by name only) make the test invalid. Pairs that depend on each other, such as agents on one team sharing a queue, make the standard error too small. And a before-and-after design cannot separate the change you made from everything else that changed in the same weeks.
In code
| What it computes | Python (scipy, statsmodels) | Excel |
|---|---|---|
| The paired t test (after minus before) | ttest_rel(calls_post, calls_pre) |
=T.TEST(C2:C81, B2:B81, 2, 1) |
| The same test as a one-sample t test on the differences | ttest_1samp(calls_post - calls_pre, 0) |
=T.DIST.2T(ABS(AVERAGE(D2:D81)/(STDEV.S(D2:D81)/SQRT(80))), 79) |
| The wrong analysis, for comparison | ttest_ind(calls_post, calls_pre, equal_var=False) |
=T.TEST(C2:C81, B2:B81, 2, 3) |
| The rank-based check on the differences | wilcoxon(calls_post, calls_pre) |
=RANK.AVG(ABS(D2), ABS($D$2:$D$81)) |
ttest_rel, ttest_1samp, ttest_ind and wilcoxon are in scipy.stats. In Excel, column D holds each agent's difference, C2 − B2; T.TEST's last argument 1 means paired. Excel has no Wilcoxon test: its route starts from the ranks of the absolute differences (chapter 14).
Quiz
Three questions on the chapter. Your best score is kept in this browser.