Exercises

The courier promises that 92 % of parcels arrive on time. Next month, 180 of 200 are on time. How often would the null world (the promise holds) produce a month at least this bad? Give P(X ≤ 180) to four decimals.

The exact share of null months at least as bad as 176 on time is P(X ≤ 176). Out of 10,000 months in the null world, about how many would be at least that bad? Give a whole number (within 1).

A manager writes: "The p-value is 0.0306, so there is only a 3 % chance that the courier kept its promise." What is wrong with that sentence?

A courier claims that deliveries take at most 2 days on average. You suspect they take longer. Which hypotheses test your suspicion?

A bank will switch suppliers only if a new card reader is faster than the old one's mean of 3.1 seconds; a reader that is slower or the same is turned down alike. Which test fits, decided before the trial?

A report on the 25 bottles, which averaged x̄ = 504 ml, states: "H₀: x̄ = 500, Hₐ: x̄ ≠ 500." What is wrong?

An analyst always uses α = 0.01 but picks the tail after seeing which way the data lean. If H₀ is true, what is the real false-alarm rate? Four decimals.

A line should fill 750 ml bottles, and its history says σ = 6 ml. 36 bottles average 747.8 ml. What is the two-tailed p-value of the z test of H₀: μ = 750? Four decimals.

A right-tailed z test at α = 0.01 gives z = 2.2. Its critical value is 2.3263. What is the p-value? Four decimals. Do the two routes agree?

A two-tailed test of whether mean delivery time differs from 2 days gives p = 0.21, and the 95 % interval for the mean runs from 1.8 to 2.9 days. Which report is best?

A slide says: "With 500,000 visitors per arm, a 0.01-point difference in conversion (4.00 % against 4.01 %) reaches p < 0.001." Run the two-proportion z test on 20,000 of 500,000 against 20,050 of 500,000. What is the two-tailed p-value? Four decimals.

A filling-line sensor raises a drift alarm on a day when the line is in fact on target. Which cell of the 2 × 2 table is that?

A team's test has too little power. They cannot change α, and the effect they care about is fixed by the business. Which lever is left, and usually the most effective?

With σ = 10 ml known, what is the power of a two-sided z test at α = 0.05 on 50 bottles to detect a real drift of 4 ml? Give it to three decimals.

A screening test decides which of 5,000 compounds go on to a cheap second assay. A false positive costs one cheap re-test; a false negative loses a possible drug. Which α suits it best?

A line should fill 750 ml bottles; σ = 6 ml is known. 36 bottles average 747.8 ml. What is the lower end of the 95 % z interval for the mean? Two decimals.

For the same 36 bottles the 95 % z interval runs from 745.84 to 749.76 ml. Without working out a p-value, what does a two-sided z test of H₀: μ = 750 at 5 % decide?

A report says: "The test and the interval can never disagree: they are the same computation." For which of these is that not guaranteed?

A second line should also fill 500 ml. An inspector weighs 16 bottles: x̄ = 503.2 ml, s = 6 ml. What is the two-tailed p-value of the one-sample t test of H₀: μ = 500? Four decimals.

For the same 16 bottles (df = 15), what is the two-tailed critical value of t at α = 0.05? Three decimals.

Back to the tutorial's bottles, x̄ = 504 ml and s = 10 ml, but 36 bottles instead of 25. What is the two-tailed p-value now? Four decimals.

Two reports. The first failed to reject and says: "The power at the observed effect, d = 0.40, was 0.484, so the result is inconclusive and we need not act." The second rejected and says: "The power at the observed effect was 0.98, so this is not a marginal finding." What is wrong with both?

A power calculation for d = 0.40 at α = 0.05 and 80 % power returns 51.009 bottles. A plan says "roughly 51 bottles would do". How many bottles does the plan need? A whole number.

Two depots deliver the same parcels. Depot A: 40 deliveries, mean 3.2 days, s = 0.9 days. Depot B: 50 deliveries, mean 2.8 days, s = 1.3 days. What is Welch's t statistic for A minus B? Two decimals.

For the same two depots, what are Welch's degrees of freedom? Two decimals.

Two call scripts were tried on five calls each. Customer satisfaction scores: script A 12, 15, 11, 14, 13; script B 10, 12, 9, 11, 13. What is Cohen's d (A minus B, with the pooled standard deviation)? Three decimals.

A readout says: "Levene's test rejected equal variances (p = 0.009), but the ratio of the standard deviations is below 2, so we kept the pooled t test as the main analysis." What should it have done?

Five stores recorded weekly returns before and after a new returns policy. Before: 52, 48, 60, 55, 58. After: 49, 47, 55, 54, 53. What is the paired t statistic (after minus before)? Two decimals.

Analyse the same five stores as if before and after were two independent groups (Welch's t test). What two-sided p-value do you get? Four decimals.

A bank measures loan processing times at 30 branches in March, installs new software everywhere, and measures the same 30 branches in May. Which test fits the question "did processing times change?"

A slide says: "After Portal v2 went live, agents made 3.89 fewer calls a day (paired t test, p = 1.5 × 10⁻⁸). Portal v2 therefore cut call volumes, and we can reduce headcount accordingly." What is wrong?

An insurer promises that 93 % of claims are settled within 30 days. Of 200 claims last month, 180 were. What is the two-sided p-value of the one-proportion z test? Four decimals.

For the same 180 of 200 claims, 14 claims are expected late (200 × 0.07). Is the normal approximation safe, and what does the exact binomial test say? Give the exact two-sided p-value, four decimals.

Meridian had 171 large losses in 4,663 claims. What is the lower end of the exact (Clopper–Pearson) 95 % interval for the large-loss rate, in per cent? Two decimals.

A newsletter tested two sign-up forms on 300 visitors each: 45 signed up with form A and 30 with form B. What is the two-proportion z statistic (A minus B)? Two decimals.

For the same two forms, what is the lower end of the 95 % Wald interval for the gap (A minus B), in percentage points? Two decimals.

What is Cohen's h for 15 % against 10 %? Three decimals.

Helion's H1 as a 2 × 2 table (portal by paperless or not). The two-proportion z statistic is 4.095. What χ² does chi2_contingency report with correction=False? Two decimals. (With its default Yates correction it reports 16.62.)

In Helion's table, 975 customers are Corporate, 2,874 pay by card, and there are 12,000 customers in all. How many Corporate card payers would independence predict? Two decimals.

A shop plans on 50 % card, 30 % contactless and 20 % cash. Of 240 customers, 110 paid by card, 80 contactless and 50 cash. What is the goodness-of-fit χ²? Three decimals. (How many degrees of freedom does it have?)

Corporate card payers: 74 observed against 233.51 expected. What is the cell's standardised residual? Two decimals.

A survey of 150 customers by plan (rows: 60 basic, 90 premium) and preferred channel (columns: app, web, phone) gives the table [[30, 20, 10], [20, 30, 40]]. The χ² test of independence gives χ² = 16.67. What is Cramér's V? Three decimals.

A dosing pump should deliver with σ no more than 2 ml. Twenty doses have s = 2.6 ml. Assuming the doses are normal, what is the one-sided p-value of the χ² test of H₀: σ = 2 against Hₐ: σ > 2? Four decimals.

Parts from supplier X (16 parts) have s = 4.1 µm; from supplier Y (21 parts), s = 2.9 µm. What is the two-sided p-value of the F test of equal variances? Four decimals.

A report compares the spread of delivery times at two depots, 200 deliveries each. The times are skewed to the right: most arrive in 2 days, a few take 10. It says: "Sample variances follow a chi-square distribution, and with 200 per depot the sample is large, so the F test is valid here." What is wrong, and what should it use?

Meridian's 95th percentile of claim size is £17,190 on telematics policies and £23,502 on standard ones. What is the ratio, telematics over standard? Two decimals. What does it say?

Three groups of 10 customers have mean days to payment of 12, 15 and 18, each with a standard deviation of 4 days. What is the one-way ANOVA's F statistic? Four decimals.

For the same three groups (between sum of squares 180, within 432), what share of the total variation lies between the groups? Give η² to four decimals.

Five regions give 10 pairs. If each pair were tested with its own t test at α = 0.05, and the tests were independent, what would be the chance of at least one false positive? Four decimals.

A report says: "Tukey's HSD cannot be used on Helion's regions, because it needs equal group sizes, and the regions have 2,365 to 3,600 customers." What is wrong?

An executive summary says: "East lags the other regions by 2.3 days." Tukey's intervals give East's gap to North, South and West as 2.35, 1.46 and 2.15 days. What should it say?

Group A's values are 3 and 5; group B's are 8, 9 and 10. List every way of choosing which two of the five values are group A's. In what share of them is A's mean minus B's mean at least as low as the real −5? This is the exact one-sided permutation p-value.

Office A's settlement times are 2.1, 3.4, 5.0 and 7.7 days; office B's are 4.2, 6.3, 8.8, 9.1 and 12.5. Rank all nine together (1 = fastest). What is U for office A, its rank sum minus n₁(n₁ + 1)/2?

Meridian's pricing team asks whether telematics lowered the average claim cost the tariff pays out. Claim amounts are extremely skewed, and there are over 2,000 claims per tier. Which test answers their question?

Six pairwise tests give p-values 0.001, 0.004, 0.012, 0.03, 0.2 and 0.5. What is the Holm-adjusted p-value of the third smallest, 0.012? Four decimals.

A limitations section says: "Region explains a few percent of claim severity (ε² = 0.0085)." What is wrong?

Two fraud models each scored the same 1,100 investigated claims as right or wrong. Is one of them right more often?

A contact centre planned for 46 % phone, 24 % chat, 18 % email and 12 % self-serve. This quarter it logged 742, 388, 271 and 199 contacts. Does the mix match the plan?

Telematics policies had 2,062 claims over 23,672 vehicle-years; standard policies 2,601 over 23,561. Policies were insured for different lengths of time. Is the claim rate lower on telematics?

Each of 30 employees' weekly output was measured the month before a training course and the month after. Did output change?

Across 20 stores, monthly footfall and sales correlate at r = 0.6. What is the t statistic for H₀: ρ = 0? Four decimals.

An analysis of 40 claim amounts says: "The data can be treated as normal, since n > 30." What is wrong?

A test on 400,000 users per arm finds a lift of 0.2 points, with a 95 % interval from 0.1 to 0.3 points and p < 0.001. The smallest lift worth acting on is 1 point. How should it be reported?

A change is rolled out to 2 million customers, and each customer it converts saves €5 a year. The 95 % interval for the lift runs from 0.8 to 2.4 percentage points. What is the annual benefit, in euros, at the interval's lower bound?

A lift is significant, with a 95 % interval from 1.2 to 4.1 points. Finance will fund the change only for a lift of at least 3 points. What is the honest summary?

A tutorial says: "At 2 million visitors, conversion of 4.0000 % against 4.0004 % gives p < 0.001." Run the two-proportion z test on 40,000 of 1,000,000 against 40,004 of 1,000,000 (2 million in all). What is the two-tailed p-value? Four decimals.

A risk review finds telematics claim severity equivalent to standard on the log scale (the geometric-mean ratio's 90 % interval, 0.871 to 0.993, is inside the ±0.20 margin) and writes: "the tariff can treat severity as unchanged". What is wrong?

A sign-up page converts 20 %. How many customers per arm does a two-sided two-proportion z test at α = 0.05 need for 80 % power against a lift to 23 %? Give a whole number (rounded up).

A two-sample t test will have 400 customers per arm. What is the smallest Cohen's d it detects with 80 % power at α = 0.05, two-sided? Three decimals.

The inspector plans to weigh 40 bottles and test H₀: μ = 500 two-sided at α = 0.05 with a one-sample t test. What is the power against a 4 ml drift, if σ is about 10 ml (d = 0.4)? Three decimals.

A report says: "p = 0.30, but the observed power was only 18 %, so the study was underpowered and the effect is probably real." What is wrong?

The power solver returns 51.009 bottles for 80 % power against d = 0.4, and a report says "roughly 51 bottles would have been needed". What is the power with 51 bottles? Four decimals.

Eight metrics give p-values 0.001, 0.008, 0.039, 0.041, 0.042, 0.06, 0.074 and 0.205. How many does Benjamini–Hochberg reject at α = 0.05?

Five metrics give p-values 0.004, 0.011, 0.013, 0.2 and 0.5. Bonferroni at α = 0.05 rejects one (only 0.004 is under 0.01). How many does Holm reject?

A dashboard tests 15 independent metrics at α = 0.05 each after every experiment. If nothing works, what is the chance that at least one comes up significant? Four decimals.

An experiment's primary metric shows nothing. The analyst slices it by region, then by segment, then by tenure, and finds p = 0.03 for new customers in the North. What is that finding?

An analysis plan reads: "multiplicity_method: fdr_bh (controls the family-wise error rate at 5 %)". What is wrong?

A governance note says: "Checking a test ten times during collection pushes a nominal 5 % false-alarm rate above 20 %." In the lab's 20,000 simulated A/A tests, each checked ten times, what share crossed 0.05 at least once?

Finance hopes Portal v2 raises paperless adoption. Before the data are opened, which alternative should the pre-registration give H1?

A join fans out and 300 customers appear twice in the experiment table. Nobody notices. What happens to H2's test?

Helion randomised 50/50 and got 5,969 customers on the old portal and 6,031 on Portal v2. What is the χ² goodness-of-fit statistic against 6,000 each? Three decimals.

A readout says: "We gate SRM at α = 0.001, not 0.05, because missing a genuine mismatch is the expensive error, so the threshold moves to protect against it." What is wrong?

H1's 95 % interval for the lift runs from 1.87 to 5.31 percentage points. With 4.2 million customers and €8.40 saved per converted customer per year, what is the annual benefit at the interval's lower end, in € millions? Two decimals.

Portal v2's 6,031 customers paid in 14.36 days on average with s = 3.40; the old portal's 5,969 in 15.44 days with s = 3.48. What is Welch's standard error of the gap, in days? Four decimals.

The same 80 agents give p = 1.5 × 10⁻⁸ with the paired test and p = 0.014 when analysed as independent groups. A report says the p-values are "three orders of magnitude apart". How many orders of magnitude apart are they, to the nearest whole number?

The ANOVA across Helion's four regions is overwhelmingly significant. Which pair of regions does Tukey's HSD not distinguish?

Helion has 8,365 Residential customers and 572 customers who pay in cash, out of 12,000. If segment and payment method were independent, how many Residential cash payers would you expect? Two decimals.

The channel-mix test gives χ² = 1.349 on 3 degrees of freedom, p = 0.717. Which sentence belongs in the capacity meeting?

H1's raw p-value is 4.22 × 10⁻⁵ and the family has six hypotheses. What is its Bonferroni-adjusted p-value? Give it to four decimals.

The next experiment must detect finance's 3-point lift from a 34 % baseline with 80 % power at α = 0.05, two-sided. How many customers per arm? A whole number.

A draft memo says: "Regional variation: East lags the others by roughly 2.3 days." What should it say?

The memo's financial sensitivity reads: lower bound €661,398, point estimate €1,267,564, upper bound €1,873,729. Which figure should the business case be underwritten at?

A memo's caveats read: "Six hypotheses were tested; Benjamini–Hochberg holds the family-wise error rate at 5 %." What is wrong?

Telematics had 2,062 claims over 23,672 vehicle-years; standard had 2,601 over 23,561. What is the rate ratio, telematics ÷ standard? Three decimals.

Why does the review compare claims per vehicle-year rather than the share of policies that had a claim?

On the same 1,100 claims, the champion alone was right on 5 and the challenger alone on 54. What is McNemar's continuity-corrected χ²? Two decimals.

Both fraud models scored the same 1,100 claims. Why is a two-proportion z test on their accuracies the wrong test?

The notebook's pre-registration lists H4 as two-sided while its alternative reads "the challenger is more often correct". What is the consistent fix?

Kruskal–Wallis on severity by region gives H = 42.6 with k = 4 regions and N = 4,663 claims. The notebook's limitations call the effect "a few percent". What is ε² = (H − k + 1)/(N − k)? Four decimals.

171 of the 4,663 claims exceed £25,000. The tariff assumes 2.5 %. What is the observed large-loss rate, as a percentage to two decimals?

TOST on log amounts shows the geometric mean ratio is within ±22 % (90 % interval 0.871 to 0.993). The notebook concludes: "the tariff can treat severity as unchanged". Why does that not follow?

The notebook's summary says "underwrite at the lower bound" but prints the saving at the point estimate, £60.6m. The rate ratio's interval runs up to 0.8360. What is the gross saving at that bound, in £ millions? One decimal.

To detect a 10 % frequency reduction from 0.1104 claims per vehicle-year with 80 % power at α = 0.05, how many vehicle-years per arm are needed? A whole number, rounded up.

A mid-term adjustment duplicates some policy rows, so a few policies show 1.4 vehicle-years on an annual policy. No single number looks wrong. What does it do to the review?

The cohort-balance check on exposure gives p = 0.116 and on the region mix p = 0.666, both at α = 0.01. What do they tell the review?

The telematics book had 0.0871 claims per vehicle-year and an average claim of £4,730. What is its loss cost per vehicle-year, in pounds? Whole pounds (within 1).

On the same claims, Welch on log amounts gives p = 0.069, Mann–Whitney p = 0.148 and Welch on the raw amounts p = 4.3 × 10⁻⁵. Which reading is right?

On the log scale, standard claims have a standard deviation of 1.4122 and telematics claims 1.3038. What is the variance ratio, standard ÷ telematics? Three decimals.