Bland–Altman Agreement

Evaluation metrics · part 4Do two models agree?

Bland–Altman analysis compares two ways of measuring the same thing, using their bias and their limits of agreement.

Same 100 emails as parts 1 to 3. Model A is the spam model from part 1. Model B is a smaller, cheaper model that someone wants to swap in. Both give every email a probability of spam. Number the emails i = 1 to 100:

Ai = model A's probability that email i is spam
Bi = model B's probability that email i is spam

Without the i, A and B mean all 100 scores from each model: two lists, lined up email by email. The question is not which model is more accurate. It is whether B gives the same scores as A, close enough that you could use either one. Bland–Altman answers that without looking at the true labels at all.

The sliders build model B from A: a constant offset, an offset that grows with the score, and random scatter. They set how B is made. The bias and limits under the plots are then measured from the 100 pairs of scores, so they land near the slider values but not exactly on them.

B against Ar =
Bland–Altman ploty = B − A
  • spam email
  • not spam
  • outside the limits

Each dot is one email. Point at one to read its two scores.

Bias
SD of differences
Limits of agreement
Outside the limits
Trend of difference
Correlation r
Labels at t = 0.50
B says 0B says 1
A says 0
A says 1
01

Correlation is not agreement

The usual first move is to plot B against A and report the correlation. Move the offset slider. Every email's B score shifts, the cloud leaves the dashed line where B = A, and r barely moves. Correlation measures whether the points lie on some straight line. Agreement needs them on one particular line, B = A. Two models can correlate at 0.99 and still disagree on every email.

02

The Bland–Altman plot

Bland and Altman's fix is to plot what you care about directly. Each dot is one email i, placed at:

x = (Ai + Bi) / 2    the best guess of email i's true score
y = Bi − Ai          how far the two models disagree on it

Perfect agreement puts every point on the horizontal line y = 0. The x axis uses the mean of the two, not A alone, because neither model is the reference. Plotting against A would build a false trend into the picture.

03

Bias

Bias is the average difference between the two models.

Subtract the scores email by email to get 100 differences, Bi − Ai, one for each dot. The bias is their average, so here A and B are the full lists of scores, not one email's:

bias = mean(B − A) = (1/n) · Σ (Bi − Ai) =

The sum runs over all n = 100 emails, and the result equals mean(B) − mean(A). It is the solid line. A positive bias means B scores emails higher than A on average. Bias is systematic, so it can be fixed: subtract it from B's scores. Whether it is real or just noise can be checked with its 95% confidence interval, bias ± 1.96 · SD / √n. If that interval excludes 0, the offset is real.

04

Limits of agreement

The limits of agreement are the range that holds about 95% of the differences.

LoA = bias ± 1.96 · SD(B − A) =

SD(B − A) is the standard deviation of the same 100 differences. The limits are the dashed lines. Read them as a promise about a new email: B's score will usually land between and of A's. The 1.96 comes from the normal distribution, so check that the differences look roughly symmetric around the bias. About 5 in 100 points should fall outside; here do. Turn up the scatter and the limits widen, even if the bias stays put.

05

Good enough?

The plot does not say whether the limits are acceptable. You decide that before looking, from what the scores are used for. For a classifier, the place a difference matters is the threshold. An email A scores 0.47 and B scores 0.53 changes label, while 0.10 against 0.16 changes nothing.

The table under the plots counts this at t = 0.50. The diagonal is where A and B give the same label, so it reads like the confusion matrices in parts 1 to 3, with A in place of the truth. Right now they disagree on of 100 emails. If those limits would flip labels you care about, B cannot replace A as it is.

06

Proportional bias

Move the middle slider. Now B drifts further from A as the score rises, and the cloud tilts. A single flat bias line hides this: it is too high at one end and too low at the other, and the limits get wider than they should. The trend of difference tile gives the slope of a straight line fitted to the points. A slope far from 0 means the agreement depends on the score, so report the bias and limits as lines that slope with the mean, or as ratios on a log scale.

07

In code

by_hand.py

import numpy as np

# a, b: arrays of 100 scores. a[i] and b[i] are model A's and model B's
# probability of spam for email i, so the arrays line up email by email.
diff = b - a          # one difference per email (y of each dot)
mean = (a + b) / 2    # one mean per email (x of each dot)

bias = diff.mean()
sd = diff.std(ddof=1)
lower, upper = bias - 1.96 * sd, bias + 1.96 * sd

outside = np.mean((diff < lower) | (diff > upper))   # about 0.05
slope = np.polyfit(mean, diff, 1)[0]                   # proportional bias
ci = 1.96 * sd / np.sqrt(len(diff))                    # bias ± ci

r = np.corrcoef(a, b)[0, 1]                            # not agreement

with_statsmodels.py

import matplotlib.pyplot as plt
import statsmodels.api as sm

sm.graphics.mean_diff_plot(b, a, sd_limit=1.96)   # plots b - a against the mean
plt.show()

Next: Part 5: Cohen's kappa, how often two models give the same label, beyond what chance would give.