Precision, Recall and F1

Evaluation metrics · part 2Precision, recall and F1

Recall reads across a row. Precision reads down a column. F1 weighs the two together.

This continues Reading a confusion matrix. Same 100 emails, same tray, same threshold slider: 60 not spam (class 0) and 40 spam (class 1). Each metric below is one group of bins divided by another. Pick one to see which bins it reads.

Drag to turn. Pick a metric.

Recall
Precision
Specificity
F1
01

Recall

Out of all the samples that are truly positive, how many did the model flag as positive?

recall = TP / (TP + FN) = / 40 =

Recall reads row 1 only: the 40 emails that really are spam. It asks how much of the spam the filter catches, and it ignores everything in row 0. A filter that flags every email gets recall 1.0, so recall alone can always be gamed.

Also called sensitivity, true positive rate (TPR) or hit rate.

02

Precision

Out of all the samples the model flagged as positive, how many are truly positive?

precision = TP / (TP + FP) = / =

Precision reads column 1 only: every email the filter sent to the spam folder. It asks how far you can trust a positive prediction. A filter that flags only its single most certain email can score 1.0 while missing nearly all spam, so precision alone can be gamed too.

Also called positive predictive value (PPV).

03

Specificity

Out of all the samples that are truly negative, how many did the model correctly leave as negative?

specificity = TN / (TN + FP) = / 60 =

Specificity is recall for class 0: it reads row 0 the way recall reads row 1. Medicine usually reports the pair sensitivity and specificity, one per row. Machine learning and search usually report precision and recall, which both centre on class 1.

Also called true negative rate (TNR) or selectivity.

04

How the threshold moves them

Recall
Precision

Raise the threshold and fewer emails reach column 1. Recall can only fall or stay put: its denominator is row 1, fixed at 40, and TP can only shrink. Precision usually rises, because the emails that leave column 1 first are the least confident ones, which are mostly false alarms. It is not guaranteed: when a true positive leaves, precision dips for a step.

Lower the threshold and the reverse happens. You buy recall with precision, and the costs of FN and FP decide where to stop.

05

F1, the harmonic mean

F1 combines precision (P) and recall (R) into one number:

F1 = 2 · P · R / (P + R) = 2TP / (2TP + FP + FN) =

The harmonic mean stays close to the smaller of the two. With P = 1.0 and R = 0.1, the plain average is 0.55, which looks decent. F1 is 0.18, which correctly says the model is poor. To score well on F1, a model has to do well on both.

The second form shows what F1 reads: TP, FP and FN. It never looks at TN, which makes it useful when negatives are plentiful and uninteresting, as with spam.

06

Macro and micro, for more than two classes

Take the cat (0), dog (1) and rabbit (2) matrix from part 1. Each class gets its own precision, recall and F1, treating that class as positive and the other two as negative. Recall reads the class's row, precision its column.

Drag to turn. Pick a class.

Three F1 scores are hard to compare between models, so they get averaged. There are two common ways:

Macro F1 computes F1 for each class, then takes the plain mean. Every class counts equally, however many samples it has, so a rare class that the model handles badly pulls the score down.

Micro F1 pools the counts first: TP is the whole diagonal, and FP and FN are the whole off-diagonal. Every sample counts equally, so large classes dominate. In single-label problems each mistake is an FP for one class and an FN for another, so pooled FP equals pooled FN and micro F1 equals accuracy.

So they answer different questions. Macro asks: how well does the model do on a typical class? Use it when every class matters equally, especially rare ones such as a disease or a fraud type. Micro asks: how well does it do on a typical sample? It is the overall hit rate, and it says little about small classes.

Reading them together is the useful part. When macro is well below micro, the model is doing well on the big classes and badly on the small ones. Here they match because every class has 50 photos and the errors are spread evenly. Weighted F1 sits between the two: per-class F1 averaged by class size, which equals macro here.

07

In code

Each metric is a count divided by a count, so it takes a few lines by hand. Writing it once is the best way to remember which cells go where. The numbers in the comments are this page's emails at t = 0.5 and the cat, dog and rabbit matrix.

by_hand.py · binary

import numpy as np

y_true = np.asarray(y_true)
y_pred = np.asarray(y_pred)

tp = np.sum((y_true == 1) & (y_pred == 1))   # 35
fp = np.sum((y_true == 0) & (y_pred == 1))   # 7
fn = np.sum((y_true == 1) & (y_pred == 0))   # 5
tn = np.sum((y_true == 0) & (y_pred == 0))   # 53

recall      = tp / (tp + fn)                    # 0.88  row 1
precision   = tp / (tp + fp)                    # 0.83  column 1
specificity = tn / (tn + fp)                    # 0.88  row 0
f1 = 2 * precision * recall / (precision + recall)   # 0.85

by_hand.py · multiclass

cm = np.array([[42,  6,  2],     # rows: actual cat, dog, rabbit
               [ 8, 35,  7],     # columns: predicted
               [ 1,  9, 40]])

tp        = np.diag(cm)
precision = tp / cm.sum(axis=0)  # down each column  [0.82 0.70 0.82]
recall    = tp / cm.sum(axis=1)  # across each row   [0.84 0.70 0.80]
f1        = 2 * precision * recall / (precision + recall)

macro_f1 = f1.mean()             # 0.78  average of per-class F1
micro_f1 = tp.sum() / cm.sum()   # 0.78  pooled, equals accuracy

with_sklearn.py

from sklearn.metrics import precision_score, recall_score, f1_score, classification_report

# binary: class 1 is positive
recall_score(y_true, y_pred)                    # 0.88
precision_score(y_true, y_pred)                 # 0.83
recall_score(y_true, y_pred, pos_label=0)       # 0.88  specificity
f1_score(y_true, y_pred)                        # 0.85

# multiclass: rebuild labels from the matrix above
y_true = np.repeat(np.repeat([0, 1, 2], 3), cm.ravel())
y_pred = np.repeat(np.tile([0, 1, 2], 3), cm.ravel())

f1_score(y_true, y_pred, average=None)          # [0.83 0.70 0.81]
f1_score(y_true, y_pred, average="macro")       # 0.78
f1_score(y_true, y_pred, average="micro")       # 0.78
print(classification_report(y_true, y_pred))    # all of it in one table

Next: Part 3: ROC AUC and PR AUC, every threshold at once.