Metrics for Imbalance

Evaluation metrics · part 8Metrics that survive imbalance

When spam gets rare, accuracy flatters. Balanced accuracy and MCC do not.

So far 40 of the 100 emails were spam. Real inboxes are rarely that even. The slider below keeps the 40 spam emails and adds more non-spam, with the same model behaviour on each, until spam is 1 email in 100. The threshold stays at 0.5. Part 7 covers choosing it.

Model

Four metrics as spam gets raret = 0.50
  • accuracy
  • balanced accuracy
  • MCC
  • F1

Accuracy
Balanced accuracy
MCC
F1
01

Accuracy's blind spot

Accuracy is the diagonal over everything. As non-spam piles up, TN becomes almost the whole table, and accuracy ends up reporting how the model treats non-spam. Pick Always says 0 and slide to 1% spam. A model that has never flagged a single email scores 99% accuracy. For the part 1 model, accuracy barely moves, because the model happens to be about equally good on both rows. It still hides what goes wrong, as the next two sections show.

02

Balanced accuracy

The average of the recall on each class: how well the model does on spam, and how well on non-spam, with each class counting half.

balanced accuracy = (TPR + TNR) / 2 = ( + ) / 2 =

Each rate is computed inside its own row, so adding more non-spam does not change it. That is why its line is flat. Always says 0 gets 1.0 on non-spam and 0 on spam, so it scores 0.5, the same as random guessing, at every share of spam. With many classes it is the average of the per-class recalls, which is part 2's macro recall.

03

Matthews correlation coefficient

The correlation between the true labels and the predicted labels, computed from all four cells.

MCC = (TP·TN − FP·FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)) =

MCC runs from −1 (always wrong) through 0 (no better than chance) to +1 (perfect). Every row and column total sits in the denominator, so it is only high when all four cells look good. Always says 0 leaves a column empty, which makes MCC 0 by convention. Unlike F1, MCC uses TN, and it gives the same value if you swap which class is called positive.

04

Two questions, two numbers

With the part 1 model at 1% spam, balanced accuracy still says 0.88 while MCC drops to about 0.2. Both are right, because they answer different questions.

Balanced accuracy asks how good the model is at each class. It is a property of the model, and it does not care how common spam is. Use it to compare models across datasets with different class balance.

MCC asks how good these predictions are, at this class balance. At 1% spam, 12% of a huge pile of good email gets flagged, and the flagged set is now mostly false alarms. Part 2's precision collapses, and MCC and F1 fall with it. Use MCC when you want one number for how useful the predictions will be in a setting where spam really is that rare.

05

In code

by_hand.py

import numpy as np

tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
balanced_acc = (tpr + tnr) / 2                                   # 0.88

den = np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
mcc = (tp * tn - fp * fn) / den if den else 0.0                  # 0.75

with_sklearn.py

from sklearn.metrics import balanced_accuracy_score, matthews_corrcoef

balanced_accuracy_score(y, y_pred)   # 0.88
matthews_corrcoef(y, y_pred)         # 0.75

Next: Part 9: Bootstrap intervals, how sure each number is.