Choosing the Threshold

Evaluation metrics · part 7Choosing the threshold

Every threshold trades one mistake for the other. Put a price on each and the best one falls out.

Parts 1 to 3 moved the threshold by hand and watched the matrix change. This part picks it on purpose. Same 100 emails, same part 1 model. The two off-diagonal bins, the mistakes, are lit in the tray.

Drag to turn. Slide the threshold.

01

Price the mistakes

Give each kind of mistake a cost in whatever unit matters: minutes, money, harm. Correct answers cost nothing. The total for one threshold is then:

cost(t) = CFP · FP + CFN · FN = · + · =

For spam, a real email lost in the spam folder is worse than one spam reaching the inbox, so the defaults price a false positive at 5 and a false negative at 1. Work out the cost at every threshold and take the lowest.

Total costlowest at t =
F1 and Youden's Jhigher is better
  • total cost
  • F1
  • Youden's J
  • cost rule
02

The rule for calibrated models

If the probabilities can be trusted, there is a shortcut. Flagging an email with spam probability p risks a false positive with chance 1 − p, and leaving it risks a false negative with chance p. Flag it when the expected cost of flagging is lower:

(1 − p) · CFP < p · CFN  →  t* = CFP / (CFP + CFN) =

Equal costs give the familiar 0.5. The dashed line in the cost chart is this rule. It only lands near the true lowest point when the model is calibrated. Part 6 showed the part 1 model is underconfident, so here the rule and the lowest-cost threshold can disagree.

03

Youden's J

When nobody can put numbers on the costs, medicine often uses Youden's J:

J = TPR − FPR = sensitivity + specificity − 1

On part 3's ROC curve, J is the height of a point above the random-guess diagonal, so the best J is the point furthest above it. Choosing by J treats a miss on a spam email and a false alarm on a good one as equally bad per email in that class, whatever the class sizes.

04

Highest F1

Part 2's F1 ignores TN, so the threshold that maximises it is a good default when positives are rare and the negatives are not interesting. It usually lands below 0.5 for an underconfident model, because recall is cheap to gain there. Unlike cost, it has no knob for saying one mistake is worse than the other.

05

Where to choose it

Choose the threshold on validation data, then report results on a separate test set at that fixed threshold. Picking the threshold on the test set and then scoring on it makes every metric look better than it will be. With only 100 emails, the best threshold also moves a lot from sample to sample. Part 9 shows how much.

06

In code

by_hand.py

import numpy as np

c_fp, c_fn = 5, 1
ts = np.round(np.arange(0.01, 1.00, 0.01), 2)

def at(t):
    pred = p >= t
    tp = np.sum(pred & (y == 1)); fp = np.sum(pred & (y == 0))
    fn = np.sum(~pred & (y == 1)); tn = np.sum(~pred & (y == 0))
    return tp, fp, fn, tn

cost = [c_fp * fp + c_fn * fn for tp, fp, fn, tn in map(at, ts)]
j    = [tp / (tp + fn) - fp / (fp + tn) for tp, fp, fn, tn in map(at, ts)]
f1   = [2 * tp / (2 * tp + fp + fn) for tp, fp, fn, tn in map(at, ts)]

t_cost = ts[np.argmin(cost)]
t_j    = ts[np.argmax(j)]
t_f1   = ts[np.argmax(f1)]
t_rule = c_fp / (c_fp + c_fn)        # only if p is calibrated

with_sklearn.py

from sklearn.metrics import roc_curve, make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV

fpr, tpr, thr = roc_curve(y, p)
t_j = thr[np.argmax(tpr - fpr)]                  # Youden's J

# tune on cross-validation folds, not on the test set
tuned = TunedThresholdClassifierCV(model, scoring="f1", cv=5).fit(X_train, y_train)
tuned.best_threshold_

Next: Part 8: Metrics for imbalance, balanced accuracy and MCC.