Evaluation metrics · part 13How far off is the number?
Most of the earlier parts scored labels. A regression model predicts a number, so every prediction is off by some amount, and the metrics differ in how they add those amounts up.
A model estimates left ventricular ejection fraction (LVEF), the percentage of blood the heart's main pumping chamber pushes out with each beat, from an echocardiogram. For 100 patients a cardiologist also measured it. Number the patients i = 1 to 100:
yi = measured LVEF of patient i, in %
ŷi = the model's estimate for patient i, in %
Without the i, y and ŷ mean all 100 values, lined up patient by patient. Every metric on this page starts from the same 100 errors:
ei = ŷi − yi positive when the model estimates too high
The sliders change the model: a constant offset, how much of the real spread between patients it keeps, random scatter, set by its standard deviation (SD), and one bad scan that sends a single estimate far too low. Every number under the plots is then measured from the 100 errors, so it lands near the slider values but not exactly on them.
- measured ≤ 40%
- measured above 40%
- label flips at 40%
Each dot is one patient. Point at one to read its two values.
Size of the errors, in LVEF points
- Biasmean error, with its sign
- MAEmean absolute error
- RMSEroot mean squared error
- MSEmean squared error, points²
- MedAEmedian absolute error
- Max errorlargest absolute error
Relative, explained and associated
- MAPEmean absolute percentage error
- R²coefficient of determination
- Explained varianceR² with the bias forgiven
- Pearson rlinear correlation
- Spearman ρrank correlation
- CCCconcordance correlation coefficient
| model > 40 | model ≤ 40 | |
|---|---|---|
| actual > 40 | ||
| actual ≤ 40 |
Errors and bias
Bias is the average error, with its sign kept.
bias = mean(e) = (1/n) · Σ (ŷi − yi) =
The sum runs over all n = 100 patients. Positive and negative errors cancel, so a model can have zero bias and still miss every patient by 10 points. Bias says which way the model leans, not how far off it is. It is part 4's bias with the measurement as the reference. Move the offset slider: the bias line in the right-hand plot moves, and the scatter of the dots around it does not.
MAE, MSE and RMSE
To measure size, stop the errors cancelling: take their absolute values, or square them.
MAE = mean(|e|) =
MSE = mean(e²) =
RMSE = √MSE =
MAE, the mean absolute error, is the typical miss, in LVEF points. MSE, the mean squared error, squares the errors first, so its units are points squared and one large error counts for far more than several small ones. RMSE, the root mean squared error, takes the square root to get back to points. RMSE is never below MAE, and the gap between them, points here, grows when a few errors are much bigger than the rest.
Turn up the bad scan. One patient's estimate falls, RMSE climbs more than twice as fast as MAE, and the max error jumps. The two ends of the range get their own summaries:
median |e| = ignores the worst misses
max |e| = is only the worst miss
Choose by the cost of a miss. If one 30-point error is worse than three 10-point errors, use RMSE. If every point of error costs the same, use MAE.
Relative error
MAPE, the mean absolute percentage error, divides each error by the true value, giving a percentage.
MAPE = mean(|ei| / |yi|) =
That makes it unit-free, so it can compare models across targets on different scales. Because LVEF is itself a percentage, read it carefully: a MAPE of 10% means the errors average a tenth of the true value, not 10 LVEF points.
It has two traps. An error of 5 points is 17% of a true LVEF of 30 but only 8% of 60, so MAPE weighs the patients with the lowest values hardest, here the sickest hearts. And as yi approaches 0 the ratio blows up. sMAPE, the symmetric mean absolute percentage error, divides by the average of |yi| and |ŷi| instead, which keeps it bounded but changes what it measures. scikit-learn returns MAPE as a fraction, so 0.10 means 10%.
R² and its relatives
R², the coefficient of determination, compares the model's squared errors with those of the laziest model: one that predicts the average for everyone.
R² = 1 − Σ ei² / Σ (yi − ȳ)² =
1 means no error. 0 means the model does no better than estimating the mean LVEF, ȳ = %, for every patient. Below 0 it does worse, which is possible: despite the name, R² is not the square of anything and has no floor. Push the offset slider to either end and R² sinks, even though the scatter has not changed.
explained variance = 1 − Var(e) / Var(y) =
Explained variance uses the variance of the errors instead of their mean square, which forgives the bias. The two differ by exactly the bias: explained variance − R² = bias² / Var(y) = . When they disagree, the model is offset rather than noisy.
adjusted R² = 1 − (1 − R²) · (n − 1) / (n − p − 1)
For a linear model fitted to the data, every extra input raises R², even a useless one. Adjusted R² charges for each of its p inputs. With n = 100 and this R², a linear model with 5 inputs would score and one with 40 would score . It is for comparing linear models of different sizes; for a network reading echo videos, p has no useful meaning.
Correlation is not R²
Pearson's correlation coefficient, r, asks whether the estimates rise and fall with the truth. It does not ask whether they are right.
r = cov(y, ŷ) / (σy · σŷ) =
Here cov is the covariance and σ the standard deviation. Move the offset slider: every estimate shifts, MAE and R² get worse, and r stays put. Now set the scatter to 0. The dots fall on a straight line and r is 1, however far the spread slider shrinks them toward the average, yet R² only reaches 1 when that line is ŷ = y. Correlation is blind to offset and scale, exactly as in part 4.
So R² can never exceed r². r² is the R² you would get after the best straight-line correction of the estimates; R² scores them as they are. Here r² = and R² = .
Spearman's rank correlation, ρ (rho), is r computed on ranks: it only asks whether a higher measurement gets a higher estimate. Turn up the bad scan and r drops much faster than ρ. A 40-point miss is a big change to the sums r is built from, but only a change of place in the ranking. Here ρ sits below r because most patients are crowded between 50% and 65%, where a few points of scatter is enough to shuffle their order.
Lin's concordance correlation coefficient, CCC, fixes the blind spot. It scores how close the dots sit to the line ŷ = y:
CCC = 2 · cov(y, ŷ) / (σy² + σŷ² + (ȳ − mean(ŷ))²) =
Offset, shrinking and scatter all pull it down. It is to correlation what Bland–Altman is to a scatter plot: agreement, not association.
Which to report
Report an error in the target's own units, so a reader can judge it: MAE, or RMSE when big misses matter more. Put the bias next to it, because a model that is consistently 5 points high is easier to fix than one that is randomly 5 points off. Add R² or CCC for how much of the difference between patients the model captures. Never report correlation alone.
Then look at where the errors fall. The table under the plots applies the ≤ 40% cut-off for reduced ejection fraction to both the measurement (the actual value, as in part 1) and the estimate, with reduced as the positive class, as spam was in part 1. It is a confusion matrix again: a 6-point error matters little at 65% and decides the label at 42%. Right now the model and the measurement disagree on of 100 patients.
In code
Every metric here is a line or two of NumPy. The numbers in the comments are this page's patients with the sliders where they start.
by_hand.py
import numpy as np
# y: measured LVEF (%), y_hat: the model's estimates (%), one per patient
e = y_hat - y # one error per patient, + means too high
bias = e.mean() # +2.8 which way the model leans
mae = np.abs(e).mean() # 4.7 mean absolute error, in LVEF points
mse = (e ** 2).mean() # 35.6 mean squared error, in points squared
rmse = np.sqrt(mse) # 6.0 root mean squared error; never below MAE
medae = np.median(np.abs(e)) # 4.0 median absolute error
maxe = np.abs(e).max() # 14.5 only the worst miss
mape = np.mean(np.abs(e) / np.abs(y)) # 0.101 mean absolute percentage error, a fraction
ss_res = np.sum(e ** 2)
ss_tot = np.sum((y - y.mean()) ** 2)
r2 = 1 - ss_res / ss_tot # 0.77 coefficient of determination
ev = 1 - e.var() / y.var() # 0.82 R² with the bias forgiven
n, p = len(y), 5 # p: inputs to a linear model
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1) # 0.76
r = np.corrcoef(y, y_hat)[0, 1] # 0.91 Pearson
rank = lambda v: np.argsort(np.argsort(v)) # no ties here; otherwise scipy.stats.rankdata
rho = np.corrcoef(rank(y), rank(y_hat))[0, 1] # 0.79 Spearman rank correlation
cov = np.mean((y - y.mean()) * (y_hat - y_hat.mean()))
ccc = 2 * cov / (y.var() + y_hat.var() + (y.mean() - y_hat.mean()) ** 2) # 0.89 Lin's concordance
with_sklearn.py
from sklearn.metrics import (mean_absolute_error, mean_squared_error, root_mean_squared_error,
median_absolute_error, max_error, mean_absolute_percentage_error,
r2_score, explained_variance_score)
from scipy.stats import pearsonr, spearmanr
# truth first: r2_score and MAPE give different answers if the arguments are swapped
mean_absolute_error(y, y_hat) # 4.7
mean_squared_error(y, y_hat) # 35.6
root_mean_squared_error(y, y_hat) # 6.0 squared=False was removed in 1.6
median_absolute_error(y, y_hat) # 4.0
max_error(y, y_hat) # 14.5
mean_absolute_percentage_error(y, y_hat) # 0.101 a fraction, not a percent
r2_score(y, y_hat) # 0.77
explained_variance_score(y, y_hat) # 0.82
pearsonr(y, y_hat).statistic # 0.91 scikit-learn has no correlation
spearmanr(y, y_hat).statistic # 0.79
# bias, adjusted R² and CCC have no scikit-learn function: use the by-hand lines