Evaluation metrics · part 10The t-test family
Five tests, one idea: a difference divided by its noise.
Same 100 emails. Model A is the part 1 model, and model B is part 4's cheaper model scoring the same emails. Each test below asks one question about their scores, their labels or their AUCs. The figure shows what each test looks at and the answer it gives.
Test
- spam email
- not spam
- mean
- value tested against
- model A
- model B
- both tails beyond the observed value: the p-value
- Difference
- Standard error
- t = diff / SE
- p-value
Signal over noise
Every t-test computes the same ratio:
t = difference / standard error standard error = SD / √n
The difference is the signal. The standard error is how much that difference would wobble from one sample to the next, which is the noise. A t of 4 means the difference is four wobbles wide.
The right chart asks how surprising that would be if there were no real difference. If there were none, t would follow the bell-shaped t distribution. The p-value is the share of that bell lying at or beyond the observed t, on both sides. A small p means a t this large would rarely happen by chance. The usual cutoff is 0.05. It is a convention, not a law.
One-sample
Is the mean of one set of numbers different from a fixed value you chose in advance?
Question: model A's average score is , but only 40% of the emails are spam. If the scores were calibrated (part 6), the average would sit near 0.40. Is the gap real?
t = (mean − 0.40) / (SD / √n) = ( − 0.40) / ( / √100) = , p =
With n − 1 = 99 degrees of freedom, p is above 0.05: a mean this far from 0.40 would turn up about one time in ten by chance, so the test finds no clear gap. That fits part 6 in a useful way. The model scores low-scoring emails too high and high-scoring emails too low, and the two errors roughly cancel in the average. A test on the mean checks only calibration on average, not the whole reliability diagram.
"Not significant" does not prove the gap is zero either. The 95% interval for the mean, , still includes gaps worth caring about. With 100 emails the test can only rule out large ones.
Paired
Two measurements of the same items: is the average difference within each pair different from 0?
Both models scored the same 100 emails, so every email gives a pair. Take d = B − A for each email and run the one-sample test on d against 0:
t = mean(d) / (SD(d) / √n) = / ( / 10) = , p =
That is all a paired t-test is. It is part 4's bias with a p-value attached, and its 95% confidence interval, , is part 4's bias interval. Move B's offset slider toward 0 and watch where p crosses 0.05.
Two-sample
Two separate groups of items: are their means different?
Question: does model A give spam emails higher scores than non-spam? These are different emails, 40 and 60 of them, so nothing pairs up. Compare the two means, with each group contributing its own noise:
t = (meanspam − meannot) / √(sspam²/40 + snot²/60) = , p =
This is Welch's version. It does not assume the groups have the same spread, and it adjusts the degrees of freedom to suit ( here). The older Student's version pools the two variances and assumes they are equal. Welch is the safer default.
The wrong test
Pick Paired data, wrong test. It takes the same A and B scores but treats them as two unrelated groups of 100, as if they came from different emails. The difference in means is identical. The answer is not:
The two tests divide the same difference by different noise. The two-sample test uses the spread of the scores, about , because some emails are obviously spam and some obviously not. The paired test uses the spread of the differences, about , because B tracks A closely on each email. Pairing cancels the email-to-email variation, so a small, consistent shift becomes visible. Ignoring the pairing throws away that information. The rule: if the same items appear in both measurements, use the paired test.
Right or wrong: McNemar's test
Two classifiers labelled the same items: is one right more often than the other?
A deployment decision usually compares labels, not scores. At threshold 0.5, model A gets of the 100 emails right and model B gets . Pick McNemar in the figure to see the 2×2 table of who got each email right. It is the paired test for right-or-wrong answers.
Emails both models got right, or both got wrong, say nothing about which model is better, so McNemar drops them. That is pairing again, the same way the paired t-test drops the email-to-email variation. Only the two disagreement cells count:
b = A right, B wrong = c = A wrong, B right =
χ² = (b − c)² / (b + c) = , p =
If the models were equally good, each disagreement would be a coin flip, so b and c would be close. With few disagreements, under about 25, use the exact version, a binomial test of b out of b + c: here p = . The trap from section 05 has a version here too. Comparing the two accuracies as if they came from different emails gives p = , because it ignores that most emails were labelled the same by both models.
Comparing AUCs: DeLong's test
Two models scored the same items: is one ROC AUC really higher than the other?
Model A's ROC AUC is and model B's is . Pick DeLong to see both ROC curves. As with McNemar, the two AUCs come from the same emails, so their errors move together, and the test has to account for that.
DeLong's method uses part 3's view of AUC as ranking pairs. For each spam email it records the share of non-spam emails ranked below it, and for each non-spam email the share of spam emails ranked above it. Doing this for both models gives each AUC's variance and, more importantly, the covariance between them:
z = (AUCB − AUCA) / √(varA + varB − 2·covAB) = , p =
The covariance term is the pairing. Here the two AUCs are strongly correlated (r = ), so the difference is far less noisy than either AUC alone. Turn up B's scatter to make B worse and watch z grow. A constant offset changes nothing, because shifting every score by the same amount does not change the ranking. Part 9's paired bootstrap of the AUC difference is the assumption-light alternative.
Assumptions and alternatives
Normal enough. The test assumes the mean, not each value, is roughly normal. With 100 items the central limit theorem usually covers that. With 10 heavily skewed values it may not.
Independent items. Each email (or each pair) must not depend on the others. Many scores from the same sender would break this.
Report the size, not just p. With enough data a trivial difference gets a tiny p. Give the difference with its 95% interval, difference ± t0.975 · SE, and let the reader judge whether it matters.
When the assumptions fail, use a rank test: Wilcoxon signed-rank for pairs, Mann–Whitney U for two groups. Mann–Whitney U divided by n1·n2 is exactly part 3's ROC AUC. Or use part 9's bootstrap, which makes no normality assumption at all.
In code
by_hand.py
import numpy as np
from scipy import stats
def one_sample(x, mu):
n = len(x)
se = x.std(ddof=1) / np.sqrt(n)
t = (x.mean() - mu) / se
p = 2 * stats.t.sf(abs(t), df=n - 1) # both tails
return t, p
one_sample(a, 0.40) # one-sample
one_sample(b - a, 0.0) # paired = one-sample on differences
def welch(x, y):
vx, vy = x.var(ddof=1) / len(x), y.var(ddof=1) / len(y)
t = (x.mean() - y.mean()) / np.sqrt(vx + vy)
df = (vx + vy) ** 2 / (vx**2 / (len(x) - 1) + vy**2 / (len(y) - 1))
return t, 2 * stats.t.sf(abs(t), df)
welch(a[y == 1], a[y == 0]) # two-sample
with_scipy.py
from scipy import stats
stats.ttest_1samp(a, popmean=0.40)
stats.ttest_rel(b, a) # paired
stats.ttest_ind(a[y == 1], a[y == 0], equal_var=False) # Welch
stats.ttest_ind(b, a, equal_var=False) # the wrong test for paired data
stats.wilcoxon(b - a) # rank-based, paired
stats.mannwhitneyu(a[y == 1], a[y == 0]) # rank-based, two groups
mcnemar.py
import numpy as np
from statsmodels.stats.contingency_tables import mcnemar
right_a = (a >= 0.5) == y
right_b = (b >= 0.5) == y
table = [[np.sum(right_a & right_b), np.sum(right_a & ~right_b)],
[np.sum(~right_a & right_b), np.sum(~right_a & ~right_b)]]
mcnemar(table, exact=False, correction=False) # chi-square on b and c
mcnemar(table, exact=True) # binomial, for few disagreements
delong.py
import numpy as np
from scipy.stats import norm
def components(s, y):
pos, neg = s[y == 1], s[y == 0]
psi = (pos[:, None] > neg[None, :]) + 0.5 * (pos[:, None] == neg[None, :])
return psi.mean(axis=1), psi.mean(axis=0) # per spam email, per non-spam email
va10, va01 = components(a, y)
vb10, vb01 = components(b, y)
s10 = np.cov(va10, vb10) # 2x2: var A, var B, cov
s01 = np.cov(va01, vb01)
var = (s10[0, 0] + s10[1, 1] - 2 * s10[0, 1]) / len(va10) \
+ (s01[0, 0] + s01[1, 1] - 2 * s01[0, 1]) / len(va01)
z = (vb10.mean() - va10.mean()) / np.sqrt(var)
p = 2 * norm.sf(abs(z))
scikit-learn and SciPy have no DeLong function. R's pROC::roc.test runs it directly.
Next: Part 11: How many test samples is enough?, sizing the test set before deployment.