Evaluation metrics · part 9How sure is that number?
A score from one test set is an estimate. A confidence interval says how far it could move.
A confidence interval is a range, computed from your data, that is likely to contain the true value of what you measured. A 95% confidence interval comes from a method that, used on many different samples, captures the true value about 95% of the time.
Here, what is measured is a model's score. Every number in parts 1 to 8 came from one set of 100 emails, and a different 100 from the same inbox would give different numbers. So "accuracy 0.88" on its own hides how much it could move, while "accuracy 0.88, 95% CI 0.81 to 0.94" says the model's true accuracy, on email like this, is plausibly anywhere in that range. This part computes that range two ways: the bootstrap, which works for any metric, and a formula, which works for accuracy and other proportions.
Metric
Emails per resample
- value on the real 100 emails
- last resample
- middle 95% of resamples, the interval
The last resample: each square is one of the 100 emails, shaded by how many times it was drawn
- left out
- once
- twice
- 3+
- On the real 100 emails
- 95% interval
- Interval width
- Resamples drawn
How to read the figure
1. A resample draws emails at random from the 100, with replacement.
2. The grid shows the last resample, one square per email.
3. Each resample gives one value of the metric. The histogram counts them: a taller bar means more resamples landed there.
4. The shaded band holds the middle 95% of those values. That band is the 95% interval.
Emails per resample sets how many emails each resample draws. Keep it at 100 to match the real test set: that gives the interval you would report. 400 and 1,600 answer a what-if question, covered in section 07.
What a confidence interval is
A confidence interval has three parts:
estimate: the value measured on the data you have, here the score on the 100 emails
lower and upper bound: the range around it
confidence level: how often the method captures the true value, usually 95%
The true value is the score the model would get on all the email it will ever see. Nobody can measure it, which is why the 100 emails stand in for it. The interval is how far the true value could plausibly sit from the estimate.
What the 95% means. Imagine drawing 100 different test sets of 100 emails from the same inbox and computing an interval from each. The true accuracy does not change, but the intervals do, because each test set gives a slightly different estimate. About 95 of those 100 intervals would contain the true accuracy, and about 5 would miss it. The 95% describes how reliable the method is, not one particular interval.
How to read the width. A narrow interval means the test set pins the score down; a wide one means it does not. Width shrinks as the test set grows (section 07). It also depends on the level: a 99% interval is wider than a 95% one from the same data, because it has to capture the true value more often.
Two readings to avoid. "There is a 95% probability that the true accuracy is in this interval" describes a Bayesian credible interval, not a confidence interval: once computed, a confidence interval either contains the true value or it does not. And "95% of emails are classified correctly within this range" is wrong: the interval is about the estimate, not about individual emails.
Definitions
- Population and sample
- The population is everything the model will ever be used on: all the email that reaches the inbox. The sample is the part you can measure: the 100 test emails. A metric on the sample estimates the metric on the population.
- Percentile
- The value below which a given percentage of the data falls. The 2.5th percentile has 2.5% of the values below it; the 97.5th has 2.5% above it. The middle 95% lies between them.
- Standard deviation (SD, s)
- How spread out the individual values are. For one email's 0/1 "classified correctly" value, the SD is √(p(1 − p)): with p = 0.88 that is 0.32.
- Standard error (SE)
- How uncertain the estimate is: the SD of the estimate itself, if the test set were redrawn many times. For a mean of n values, SE = s / √n. For the 100 emails, 0.32 / √100 = 0.032. It shrinks as the sample grows.
- 95% confidence interval (CI)
- A range computed from the data such that, if the test set were redrawn many times, about 95% of the intervals would contain the true value of the metric.
- Why 1.96
- In a normal distribution, 2.5% of the area lies above +1.96 SE and 2.5% below −1.96 SE, so estimate ± 1.96 × SE covers the central 95%. The 2.5th and 97.5th percentiles of the bootstrap are the same two cut points, read off the data instead of assuming a normal shape.
The test set is a sample
The 100 emails are a sample from all the email the filter will ever see. The metric on them is an estimate, and an estimate needs an error bar. Without one, a model at 0.96 and a model at 0.94 look different even when that gap is just which emails happened to land in the test set.
To get the error bar you would ideally test on many fresh sets of 100 emails and see how much the score moves. You only have one. The bootstrap (sections 04 and 05) imitates the fresh sets using the one you have; the formula (section 06) gets the same answer for accuracy from theory.
Resampling with replacement
The bootstrap treats the 100 emails as a stand-in for the whole inbox and draws new test sets from them:
1. draw 100 emails at random, with replacement
2. compute the metric on that resample
3. repeat 1,000 times
Press Draw 1 resample and look at the grid. Some emails are picked twice or more, and on average about 37% are left out entirely: the chance that one email is never picked in 100 draws is 0.99100 ≈ 0.366, close to 1/e. Each resample is a plausible alternative test set, and its metric is one plausible alternative result.
The percentile interval
After 1,000 resamples, sort the results and cut off 2.5% at each end. What is left is the 95% bootstrap interval:
95% CI = [ 2.5th percentile, 97.5th percentile ] =
Report it with the number: "ROC AUC 0.96, 95% CI 0.93 to 0.99". Read it as the range of results this model could plausibly have produced on another test set of the same size. Switch the figure to accuracy, F1 and MCC: the interval is wider for F1 (0.76 to 0.93) and MCC (0.61 to 0.88) than for accuracy (0.81 to 0.94). F1 ignores the 53 emails correctly let through, so it rests on fewer emails, and MCC runs from −1 to 1, so the same uncertainty covers more of its scale.
The percentile method is the simplest. When the metric leans against a limit, as AUC does near 1, the percentile interval is biased; the BCa interval corrects for that skew (scipy.stats.bootstrap(..., method="BCa"), in the code below).
Without the bootstrap
Accuracy is a proportion: each email is classified correctly (1) or not (0), and accuracy is the mean of those 0/1 values. A 0/1 value has SD √(p(1 − p)), so the standard error has a formula and the interval needs no resampling. The number of classes does not matter: accuracy over four classes is still correct / n.
p = correct / n
SE = √( p (1 − p) / n )
95% CI = p ± 1.96 × SE
For the 100 emails the model gets 88 right, so p = 0.88, SE = √(0.88 × 0.12 / 100) = 0.032, and the 95% CI is 0.88 ± 0.064 = 0.816 to 0.944. The bootstrap in the figure, set to accuracy, gives 0.81 to 0.94: the same interval, without a single resample.
When it applies: accuracy and error rate, and any metric that is a mean of per-email 0/1 values. Precision and recall are too, with n the number of emails flagged as spam (precision) or the number that really are spam (recall). The test set must be large enough for the normal approximation: as a rule of thumb, n × p and n × (1 − p) both above 10. Here they are 88 and 12.
When it does not: ROC AUC, F1, MCC, correlation, and anything else that is not a mean of per-email values. Bootstrap those. AUC also has a formula-based interval (DeLong, in part 10), but the bootstrap is simpler and just as accepted.
When p is near 0 or 1, or n is small, p ± 1.96 × SE can give bounds below 0 or above 1. The Wilson interval fixes this and is the better default for proportions. For the 100 emails it gives 0.802 to 0.930. It is one line in statsmodels, and the calculator shows it.
- Accuracy p
- SE
- Normal 95% CI
- Wilson 95% CI
- Bootstrap 95% CI (5,000)
For accuracy the three agree closely; the bootstrap is there to show it. Try 98 correct out of 100: the normal interval crosses 1.0, while Wilson stays inside.
More data, narrower interval
What if the test set had 400 emails, or 1,600? Set emails per resample to 400, then 1,600. The figure redraws 1,000 resamples of that size. The interval roughly halves each time the size quadruples, because the standard error shrinks with √n. That answers "is 100 emails enough?": enough for a rough number, not enough to tell 0.96 from 0.94.
The formula gives the same answer directly. At 88% accuracy, the 95% CI is about ± 6.4 points with 100 emails, ± 3.2 with 400, ± 1.6 with 1,600, and ± 1.4 with 2,000. Part 11 turns this around and asks how many emails you need for a given width.
This is a what-if. The larger resamples still draw from the same 100 emails, so they show how the interval would shrink, not what a real larger test set would score. Only new emails add new information.
Training seeds
Everything above answers one question: how much would the score move with different test emails? Training randomness (weight initialization, data shuffling, dropout) is a second, separate question: how much would it move if the model were trained again? The two are measured differently and reported side by side.
| Source of uncertainty | Question | What you need | Method |
|---|---|---|---|
| Test-set sampling | Different test emails, same model | One trained model, n test predictions | Formula or bootstrap CI, above |
| Training randomness | Same test emails, retrained model | k models, one per seed | Mean ± SD, or a t-based CI, across seeds |
A t-based CI across seeds. Train k models with seeds 0 … k − 1, score each on the same test set, and treat the k scores as the data. The interval is mean ± t × SE, with a t-value instead of 1.96: the SD is estimated from only k values, so the t-distribution's heavier tails widen the interval to compensate. Part 10 covers the t-distribution.
CI = x̄ ± tk−1 × s / √k
x̄ = mean score across seeds, s = their SD, k = number of seeds
| Seeds (k) | Degrees of freedom (k − 1) | t for 95% |
|---|---|---|
| 3 | 2 | 4.303 |
| 5 | 4 | 2.776 |
| 10 | 9 | 2.262 |
| 30 | 29 | 2.045 |
| very many | very many | 1.960 |
- Mean
- SD
- SE = SD / √k
- t (df = k − 1)
- 95% CI across seeds
Which seed gets the test-set CI. Compute it for the seed you present as the main result: the median seed, or seed 0 if that was fixed in advance. Not the best-scoring seed, which flatters the result.
Do not pool predictions across seeds. Stacking 5 seeds × 100 predictions into one binomial CI over 500 "emails" counts the same 100 emails five times, and can make the interval up to √5 ≈ 2.2 times too narrow. A valid combined number: average each email's correctness across the seeds, then bootstrap those 100 per-email values. That is the test-set CI of the seed-averaged model, and it still does not replace mean ± SD across seeds.
How many seeds. With only 100 test emails, the test-set interval (about ± 6 points) is far wider than typical seed-to-seed variation, so more test emails help more than more seeds. With a few thousand test emails the test-set interval is tight, and 3 to 5 seeds are the better use of compute. Report seed variation whenever you claim one method beats another: it can be larger than the gap between them.
Comparing two models
To compare two models, do not check whether their two intervals overlap: two intervals can overlap even when the difference between the models is real. Use a paired bootstrap: score both models on the same resample each time and record the difference. If the 95% interval of the difference excludes 0, the gap is unlikely to be luck of the test set. Pairing matters because the same hard emails trip up both models, which makes the difference far less noisy than either metric alone.
If both models were trained with several seeds, also compare the per-seed scores with a paired t-test, pairing the same seed for both models (part 10).
How to report
Give the estimate, the interval, n, and the method. For the 100 emails: the accuracy is 88%, and the 95% confidence interval is 80.2% to 93.0%.
| Situation | Report |
|---|---|
| One model, accuracy | Accuracy = 88.0% (95% Wilson CI: 80.2%–93.0%, n = 100) |
| One model, any metric | ROC AUC = 0.96 (95% bootstrap CI: 0.93–0.99, 1,000 resamples, n = 100) |
| Several seeds | Accuracy = 87.6 ± 1.1% (mean ± SD, 5 seeds); test-set 95% CI for seed 0: 80.2%–93.0% |
| Two models | Δ accuracy = +2.0 points (paired bootstrap 95% CI: +0.5 to +3.5) |
State once which method produced each interval: normal (binomial), Wilson, bootstrap percentile, BCa, or t across seeds. The two-model row shows the format only; this page has one model.
In code
by_hand.py
import numpy as np
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(0) # seed for resampling only, not training
n, stats = len(y), []
for _ in range(1000):
i = rng.integers(0, n, n) # n draws with replacement
if y[i].min() == y[i].max(): # AUC needs both classes
continue
stats.append(roc_auc_score(y[i], p[i]))
low, high = np.percentile(stats, [2.5, 97.5]) # about 0.93 to 0.99
# paired: same i for both models, keep the difference
# diff.append(roc_auc_score(y[i], p_a[i]) - roc_auc_score(y[i], p_b[i]))
accuracy_formula.py
from statsmodels.stats.proportion import proportion_confint
correct = (y_pred == y) # True/False per email, any number of classes
k, n = correct.sum(), len(correct) # 88, 100
acc = k / n
se = np.sqrt(acc * (1 - acc) / n) # 0.032
ci_normal = (acc - 1.96 * se, acc + 1.96 * se) # 0.816 to 0.944
ci_wilson = proportion_confint(k, n, alpha=0.05, method="wilson") # 0.802 to 0.930
seeds.py
from scipy import stats
acc = np.array([0.880, 0.870, 0.890, 0.860, 0.880]) # one model per training seed
m, se = acc.mean(), acc.std(ddof=1) / np.sqrt(len(acc))
ci = stats.t.interval(0.95, df=len(acc) - 1, loc=m, scale=se) # 0.862 to 0.890
# never pool the 5 x 100 predictions into one binomial CI:
# average each email's correctness across seeds, then bootstrap those 100 values
with_scipy.py
from scipy.stats import bootstrap
res = bootstrap((y, p), lambda y, p: roc_auc_score(y, p),
paired=True, vectorized=False, n_resamples=1000,
method="BCa", random_state=0)
res.confidence_interval
Next: Part 10: The t-test family, testing whether a difference is real.