ROC AUC and PR AUC

Evaluation metrics · part 3ROC AUC and PR AUC

Each threshold gives one confusion matrix. Sweep every threshold and the matrices trace a curve.

Parts 1 and 2 read one matrix at one threshold. The metrics here summarise the model across all thresholds at once. Same 100 emails, same tray: 60 not spam (class 0), 40 spam (class 1).

Drag to turn. Slide the threshold.

01

One matrix, one point

Each curve needs two numbers per matrix. Part 2 defined recall, which ROC calls the true positive rate. ROC adds one more:

False positive rate: out of all the samples that are truly negative, how many did the model wrongly flag as positive?

TPR = TP / (TP + FN) = / 40 =
FPR = FP / (FP + TN) = / 60 =

TPR reads row 1 and FPR reads row 0. FPR is 1 minus specificity. Plot (FPR, TPR) and you get one point on the ROC curve. Plot (recall, precision) and you get one point on the precision–recall (PR) curve. The dots below are the current threshold. Press Sweep to visit every threshold from 1 down to 0, or drag along a curve.

ROC curveAUC
PR curveAUC

At t = · FPR · TPR · recall · precision

02

The ROC curve and its AUC

At a threshold above every score, nothing is flagged: TP = FP = 0, the point (0, 0). At a threshold of 0, everything is flagged: the point (1, 1). In between, lowering the threshold moves emails into column 1 one score at a time. A spam email moves the curve up, a non-spam email moves it right. A good model collects most of the spam before it collects much non-spam, so its curve hugs the top-left corner.

ROC AUC is the area under that curve, added up as trapezoids between neighbouring points. It also has a plain meaning. Pick one spam email and one non-spam email at random. ROC AUC is the probability that the model gives the spam email the higher score, with ties counting half.

03

The PR curve and its AUC

The PR curve plots precision against recall, so it only uses TP, FP and FN. It starts at the left with the few emails the model is most sure about, where precision is usually high. It ends at recall 1, where everything is flagged and precision drops to the share of spam in the data, 40 / 100 = 0.40.

Its area is usually computed as average precision (AP): the precision at each threshold, weighted by how much recall that threshold added.

AP = Σ (Rk − Rk−1) · Pk =

That is why the PR curve above is drawn in steps. Joining PR points with straight lines, the way ROC does, overstates the area, so libraries use the step sum instead. "PR AUC" in most reports means this number.

04

Random, weak and strong

An AUC means little without a baseline. Swap in a different model below. The tray and both curves above update with it.

Random guess, ROC. A model whose scores ignore the email flags spam and non-spam at the same rate, so TPR = FPR at every threshold. Its curve is the diagonal and its ROC AUC is 0.5, whatever the class balance. Below 0.5 means the scores run backwards.

Random guess, PR. Whatever a random model flags is a random sample of the emails, so 40% of it is spam at every recall. Its curve is a flat line at the share of positives, and its PR AUC equals that share: 0.40 here. If only 1 email in 100 were spam, the random PR AUC would be 0.01 while the random ROC AUC stayed at 0.5.

Weak and strong. As the model separates the two classes better, the ROC curve bends toward the top-left and the PR curve toward the top-right. A perfect model ranks every spam email above every non-spam email and scores 1.0 on both. The small sample makes each number wobble: the random model here lands near 0.5 and 0.40, not exactly on them.

05

Which one to report

ROC AUC treats both rows equally, so it reads the same whatever the class balance. That makes it easy to compare across datasets, but when positives are rare, a large pile of easy negatives keeps FPR tiny and flatters the model. PR AUC ignores TN and is read against the share of positives, so it shows how useful the flagged set is when positives are rare. When the positive class is rare and it is what you care about, as with fraud, disease or spam, report PR AUC, and always state its baseline next to it.

06

In code

by_hand.py

import numpy as np

# y: true labels (1 = spam), p: predicted probability of class 1
order = np.argsort(-p, kind="stable")
y, p = y[order], p[order]
last = np.r_[np.diff(p) != 0, True]       # last row of each tied score

tp = np.cumsum(y)[last]                    # flag everything with score >= each threshold
fp = np.cumsum(1 - y)[last]
P, N = y.sum(), (1 - y).sum()

tpr = np.r_[0, tp / P]                     # start at (0, 0)
fpr = np.r_[0, fp / N]
roc_auc = np.trapezoid(tpr, fpr)           # trapezoids, 0.96

recall    = tp / P
precision = tp / (tp + fp)
ap = np.sum(np.diff(np.r_[0, recall]) * precision)   # step sum, 0.94

# ROC AUC as a ranking probability
pos, neg = p[y == 1], p[y == 0]
diff = pos[:, None] - neg[None, :]
rank_auc = np.mean((diff > 0) + 0.5 * (diff == 0))  # same as roc_auc

with_sklearn.py

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.metrics import precision_recall_curve, average_precision_score

fpr, tpr, thresholds = roc_curve(y, p)
roc_auc_score(y, p)                         # 0.96

precision, recall, thresholds = precision_recall_curve(y, p)
average_precision_score(y, p)               # 0.94
y.mean()                                    # 0.40, the PR baseline

On NumPy older than 2.0, np.trapezoid is called np.trapz.

Next: Part 4: Bland–Altman, whether two models give the same scores.