Evaluation metrics · part 6Can you trust a 0.7?
A model is calibrated when its probabilities come true at the rate they claim.
Parts 1 to 5 used the model's probabilities to rank emails or to cut them at a threshold. Neither needs the numbers to mean anything. This part asks whether they do: of all the emails the model scores around 0.7, are about 70% spam? Same 100 emails, same part 1 model.
Bins
- ECE
- Brier score
- ROC AUC
- Scores used
The reliability diagram
Sort the emails into bins by predicted probability: 0 to 0.2, 0.2 to 0.4, and so on. For each bin, plot two numbers:
x = average predicted probability in the bin (what the model claims)
y = share of the bin that really is spam (what happened)
A calibrated model sits on the dashed diagonal. A point below it means the model claimed more spam than there was. A point above it means the model claimed less. The part 1 model makes an S shape: below the line on the left and above it on the right. Emails it scores near 0.3 are almost never spam, and emails it scores near 0.7 almost always are. The model ranks well but is underconfident: its probabilities sit too close to 0.5.
Expected calibration error
ECE turns the diagram into one number: the gap between claim and outcome in each bin, weighted by how many emails the bin holds.
ECE = Σbins (nb / N) · | share spamb − average predictionb | =
The orange lines in the diagram are those gaps. 0 is perfect. ECE depends on the bins, so switch between 5 and 10 and watch it move. With only 100 emails, 10 bins leave some nearly empty, and their gaps are mostly noise.
Brier score
The Brier score needs no bins. It is the mean squared distance between each probability and what happened (1 for spam, 0 for not):
Brier = (1/N) Σ (pi − yi)² =
0 is perfect. The baseline is a model that ignores the email and always predicts the share of spam, 0.40: its Brier score is 0.40 × 0.60 = 0.24. Always saying 0.5 scores 0.25. Unlike ECE, Brier rewards ranking as well as calibration, so a lower Brier can come from either.
Ranking is not calibration
Move the reshape slider. It pushes every probability toward 0 and 1 (right) or toward 0.5 (left) without changing their order. The diagram, ECE and Brier all move. ROC AUC does not, because part 3's AUC only depends on which email scores higher than which. The two properties are independent: a model that always says 0.40 is perfectly calibrated with AUC 0.5, and a model can have AUC 1.0 and still say 0.55 for every spam email.
Fixing it
Calibration can be repaired after training, without touching the ranking. Platt scaling fits a small logistic regression on the model's own scores:
pnew = σ(a · logit(p) + b) a = , b =
Press Fix with Platt scaling. Here a is above 1, which sharpens the scores, as an underconfident model needs. Then move the reshape slider: the fit undoes whatever you do. Isotonic regression is the other common fix. It fits any increasing step function, so it is more flexible and needs more data.
This page fits Platt scaling on the same 100 emails it then scores, which flatters the result. In practice, fit it on held-out data.
Calibration matters whenever the number itself is used, not just the ranking: showing a user a probability, adding up expected counts, or picking a threshold from costs, which part 7 does.
In code
by_hand.py
import numpy as np
# y: true labels, p: predicted probability of class 1
bins = np.linspace(0, 1, 6) # 5 equal-width bins
b = np.clip(np.digitize(p, bins) - 1, 0, 4)
ece = 0.0
for k in range(5):
m = b == k
if m.any():
gap = abs(y[m].mean() - p[m].mean()) # outcome vs claim
ece += m.mean() * gap # weighted by bin size
brier = np.mean((p - y) ** 2) # 0.11
with_sklearn.py
from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
frac_spam, mean_pred = calibration_curve(y, p, n_bins=5) # the diagram's points
brier_score_loss(y, p) # 0.11
# refit with calibration, using cross-validation to avoid the self-fit
calibrated = CalibratedClassifierCV(model, method="sigmoid", cv=5) # "isotonic" also works
calibrated.fit(X_train, y_train)
Next: Part 7: Choosing the threshold, putting a price on each mistake.