Reading a Confusion Matrix

Evaluation metrics · part 1Four bins for every prediction

A classifier can be right in two ways and wrong in two ways. A confusion matrix counts all four.

Think of the matrix as a sorting tray with four bins. Each prediction is a marble, and each marble lands in exactly one bin. The tray below sorts 100 emails for a spam filter: 60 are not spam (class 0) and 40 are spam (class 1).

Drag to turn. ← →

  • really spam (1)
  • really not spam (0)
01

From probability to label

A binary classifier outputs its estimated probability of class 1 for each input, here the probability that the email is spam. A threshold t turns that probability into a label:

p ≥ t  →  predict 1 (spam)
p < t  →  predict 0 (not spam)

Move the slider above. Raising t shifts emails from column 1 to column 0, turning false positives into true negatives but true positives into false negatives. Lowering it does the reverse. Pick t by which mistake costs more.

02

Four outcomes

Read each name backwards. The second word is what the model predicted: positive (1) or negative (0). The first word says whether that prediction was right.

True negative TN
Predicted not spam, and it is not.row 0, column 0 · now:
False positive FP
Predicted spam, but it is not. A false alarm, or Type I error.row 0, column 1 · now:
False negative FN
Predicted not spam, but it is spam. A miss, or Type II error.row 1, column 0 · now:
True positive TP
Predicted spam, and it is spam.row 1, column 1 · now:

"Positive" means class 1, the class you are looking for, not a good outcome. In a cancer screen, positive is cancer.

03

Rows and columns

Rows are the truth. Row 0 holds every email that really is not spam, row 1 every email that really is spam. Row totals come from the data, so they never change: 60 and 40.

Columns are the predictions. Column 0 holds everything labelled 0, column 1 everything labelled 1. Column totals depend on the model: right now and .

Pick a row, a column or a diagonal.

This layout (rows actual, columns predicted, class 0 first) is what scikit-learn prints. Some textbooks put class 1 first, so check the labels on any matrix you did not make.

04

The diagonal, and everything else

The diagonal runs top-left to bottom-right, through the cells where row and column match. Diagonal cells are correct predictions. Off-diagonal cells are mistakes, and each one names its mistake: the row says what the item was, the column says what the model confused it with. A perfect model has zeros everywhere off the diagonal.

Accuracy is the diagonal's share of the total, and it can hide a lot. If only 10 of 1,000 emails are spam, a filter that never predicts spam scores 99%, yet its column 1 is empty.

05

More than two classes

With k classes the matrix is k×k, and the rules stay the same. Here an image model labels 150 photos: 50 each of cat (0), dog (1) and rabbit (2). Bar height is the cell count.

Drag to turn. Point at a bar to read it.

For any one class, say dog, the four outcomes still exist. TP is its diagonal cell, FN the rest of its row, FP the rest of its column, and TN everything else. The matrix need not be symmetric: rabbits are called dogs 9 times, dogs called rabbits only 7.

06

In code

Each bin is one comparison, so the matrix takes four lines by hand. The numbers in the comments are this page's 100 emails at t = 0.5.

by_hand.py

import numpy as np

p = model.predict_proba(X_test)[:, 1]          # P(class 1)
y_pred = (p >= 0.5).astype(int)                # t = 0.5

tn = np.sum((y_test == 0) & (y_pred == 0))   # 53  row 0, column 0
fp = np.sum((y_test == 0) & (y_pred == 1))   # 7   row 0, column 1
fn = np.sum((y_test == 1) & (y_pred == 0))   # 5   row 1, column 0
tp = np.sum((y_test == 1) & (y_pred == 1))   # 35  row 1, column 1

cm = np.array([[tn, fp],
               [fn, tp]])
assert cm.sum() == len(y_test)                 # every email lands in one bin

with_sklearn.py

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)   # [[TN FP]
tn, fp, fn, tp = cm.ravel()             #  [FN TP]]

Pass labels=[1, 0] to put class 1 first, which moves TP to the top-left.

Next: Part 2: precision, recall and F1, read off the same matrices.