Evaluation metrics · part 5Cohen's kappa
How often two raters agree, minus the agreement luck alone would give.
Part 4 asked whether models A and B give the same scores. This part asks whether they give the same labels. The tray now sorts the 100 emails by two raters instead of truth and prediction: rows are model A's label, columns are model B's. Neither rater is treated as correct. The marbles keep their spam and not-spam colours from part 1, but nothing on this page uses them.
Drag to turn. Choose B, then slide the threshold.
Observed agreement
Out of all the samples, how many did the two raters label the same?
po = (both 0 + both 1) / n = ( + ) / 100 =
That is the diagonal of the tray, read the way part 1 read accuracy. It is easy to inflate. Pick Always 0 for B: a rater that never looks at the email still agrees with A on every email A calls not spam. Agreement looks respectable, and none of it means anything.
Agreement by chance
If the two raters labelled independently, each keeping its own rate of saying 1, how often would they agree by luck?
Independence means each cell's expected count is its row total times its column total, divided by n. The chance agreement is the expected diagonal:
pe = (A0·B0 + A1·B1) / n² = (· + ·) / 10,000 =
Observed counts
Expected by chance
Two raters that both say 0 most of the time will agree often by luck, so pe rises whenever both lean the same way.
Kappa
Kappa asks how much of the room above chance the raters actually used:
κ = (po − pe) / (1 − pe) = ( − ) / (1 − ) =
The bar runs from 0 to 1. The striped part is what luck gives for free. Kappa is the orange part divided by everything to the right of the stripes. κ = 1 means perfect agreement, κ = 0 means no better than chance, and κ < 0 means the raters disagree more than chance would predict.
Four kinds of B
Same A, same threshold, four different second raters:
Always 0 has the trap from section 01: decent po, yet κ = 0 exactly, because its pe equals its po. Coin flip says 1 at random 40% of the time and lands near 0. The truth turns the tray back into part 1's confusion matrix, with A as the prediction. Kappa against the true labels is a classification metric in its own right: accuracy with chance agreement removed.
The prevalence trap
Pick Cheap model and push the threshold toward 0.9. Both models now say 0 for almost every email. Observed agreement climbs, but chance agreement climbs faster, and kappa falls. The two raters have not got worse at agreeing on the emails that matter. There are just very few 1s left to agree on, so each disagreement weighs more.
This is why kappa is hard to compare across datasets with different class balance. Report it next to po and the two raters' rates of saying 1, so a reader can see which of these is going on.
Reading the number
The most quoted scale is Landis and Koch's, from 1977:
| Kappa | Agreement |
|---|---|
| below 0 | poor |
| 0.00 – 0.20 | slight |
| 0.21 – 0.40 | fair |
| 0.41 – 0.60 | moderate |
| 0.61 – 0.80 | substantial |
| 0.81 – 1.00 | almost perfect |
The bands were a judgement call, not derived from anything, and section 05 shows the same raters can move between them with the threshold alone. Use them as rough words, not as pass marks.
More than two labels
The formula does not change with k labels. po is still the diagonal share of a k×k table, and pe is still the sum over labels of (row total × column total) / n². When labels are ordered, such as severity 1 to 5, plain kappa treats "off by one" the same as "off by four". Weighted kappa fixes that by giving partial credit to near misses, with linear or quadratic weights.
In code
by_hand.py
import numpy as np
a = (p_a >= 0.5).astype(int) # model A's labels
b = (p_b >= 0.5).astype(int) # model B's labels
n = len(a)
table = np.zeros((2, 2))
np.add.at(table, (a, b), 1) # rows: A, columns: B
p_o = np.trace(table) / n # 0.93
p_e = table.sum(axis=1) @ table.sum(axis=0) / n**2 # 0.50
kappa = (p_o - p_e) / (1 - p_e) # 0.86
with_sklearn.py
from sklearn.metrics import cohen_kappa_score
cohen_kappa_score(a, b) # 0.86, two raters
cohen_kappa_score(y_true, a) # model A against the truth
cohen_kappa_score(r1, r2, weights="quadratic") # ordered labels
Next: Part 6: Calibration, whether a 0.7 really means 70%.