Evaluation metrics · part 11How many test samples is enough?
Enough that the number you report is precise enough to make the decision, and no fewer.
Every part of this series used 100 emails, 40 of them spam. Part 9 showed that is enough for a rough number but not for a fine one. The spam filter's recall of 0.88 is really 0.88 ± 0.10. Before deploying a model, you work the other way: decide how precise the number must be, then work out how many samples get you there. This is called sample size calculation.
Write the requirement first
Start with one sentence, written before you see any results, for example: "Recall on spam must be at least 0.85, and we expect the model to reach about 0.90." It fixes three things the calculation needs: which metric, which threshold it must clear, and what you expect it to be. Deciding these after seeing the results lets the data choose the bar.
Margin of error
Recall, precision, specificity and accuracy are all proportions: a count of successes out of n tries. Their 95% interval is roughly
rate ± E, E = 1.96 · √( p(1 − p) / n ) → n = 1.96² · p(1 − p) / E²
E is the margin of error, p is the rate you expect, and n is the number of samples the rate is computed on. Choose the margin you can live with and the formula gives n. This is precision-based sample size: it sizes the test set so the estimate is tight enough to report.
- Positives needed
- Total test samples
- With this series' 40 spam emails
The curve flattens fast. Going from 100 to 400 positives halves the margin, but going from 400 to 1,600 only halves it again: the margin shrinks with √n. The hardest cases are rates near 0.5, where p(1 − p) is largest. Rates near 1 need fewer samples for the same margin.
Count the right samples
n in the formula is the number of samples the metric is computed on, not the size of the test set. Part 2 showed which cells each metric reads:
- Recall reads row 1, so n is the number of true positives in the test set.
- Specificity reads row 0, so n is the number of true negatives.
- Precision reads column 1, so n is the number the model flags.
Then divide by the share of positives to get the total. Move the share slider above to 5%: 385 positives becomes 7,700 test samples. Rare positives are what make test sets large.
Pass or fail: power and effect size
A deployment decision is usually pass or fail: does recall clear 0.85? The rule is to pass only if the lower end of the 95% interval clears the bar. Recall of 0.88 with an interval of 0.84–0.92 fails a "≥ 0.85" requirement, even though 0.88 is above it.
Sizing for that decision is power analysis. It needs four numbers:
- The requirement, p0: the bar to clear.
- The expected rate, p1: what you think the model really achieves. The gap between them is the effect size, the smallest difference you need the test to detect. For proportions it is measured as Cohen's h = 2·arcsin√p1 − 2·arcsin√p0.
- α: the risk of passing a model that does not really clear the bar. 0.05 is usual.
- Power: the chance of passing a model that does clear it. 0.80 or 0.90 is usual.
n = ( (zα + zpower) / h )²
α, one-sided
Power
- Effect size, Cohen's h
- Positives needed
- Total at positives
Close the gap between p0 and p1 and the count explodes, because n grows with 1 / h². Halving the effect size roughly quadruples the samples. If the model is expected to clear the bar only barely, no affordable test set will confirm it. That is worth knowing before you collect anything.
The simulation button checks the formula. It draws 1,000 test sets of the computed size from a model whose true recall is p1, runs the pass/fail test on each, and counts how often the model passes. That share should land near the chosen power.
When you expect no failures
Some requirements are about rare failures, for example "fewer than 1 in 1,000 legitimate emails blocked". If the test shows zero failures in n samples, the 95% upper bound on the failure rate is about 3 / n. This is the rule of three. To support a claim of "below r", you need about 3 / r clean samples.
Clean samples needed
A single failure moves the bound up to about 4.7 / n, so it pays to test with some margin to spare.
Every subgroup counts
If the requirement must hold for each class, each region or each user type, every one of them needs its own count. The smallest subgroup sets the size of the whole test set. A rare language that is 2% of traffic needs its 385 positives just as much as the main group does. For the many-class case, this is part 2's macro averaging seen from the planning side: each class needs enough samples to be measured on its own.
Make the test set count
A large test set does not help if it is the wrong one.
- Representative: same sources, time period and class mix as production. If you oversample rare positives to reach the count, re-weight precision back to the real prevalence (part 8).
- Untouched: never used for training, model selection or choosing the threshold (part 7).
- Fixed in advance: metric, requirement, threshold, α and power written down before the test is run.
- Reported with intervals: every number with its 95% interval (parts 9 and 10), and the decision made on the lower bound.
In code
by_hand.py
import numpy as np
from scipy.stats import norm
# precision-based: positives needed for recall 0.90 ± 0.03
p, E = 0.90, 0.03
n_pos = norm.ppf(0.975)**2 * p * (1 - p) / E**2 # 384.2 -> 385
n_total = n_pos / 0.05 # at 5% positives
# power-based: recall must beat 0.85, expected 0.90
p0, p1, alpha, power = 0.85, 0.90, 0.05, 0.80
h = 2 * np.arcsin(np.sqrt(p1)) - 2 * np.arcsin(np.sqrt(p0)) # 0.152
n_pos = ((norm.ppf(1 - alpha) + norm.ppf(power)) / h) ** 2 # 268
# rule of three: zero failures in n -> rate below 3 / n
n_clean = 3 / (1 / 1000) # 3,000
with_statsmodels.py
from statsmodels.stats.proportion import proportion_effectsize, proportion_confint
from statsmodels.stats.power import NormalIndPower
h = proportion_effectsize(0.90, 0.85) # Cohen's h
NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.8,
ratio=0, alternative="larger") # 268, one sample
proportion_confint(count=350, nobs=385, method="wilson") # interval to report
Next: Part 12: Splitting data without leaking, keeping the test emails unseen until the end.