Machine learning from scratch · part 2Loss functions
A loss function turns one prediction and the truth into a single number. Training makes the average of that number as small as it can, so the choice of loss decides what the model learns.
12 October 2026
12 min read
23 live plots
Each card plots one loss against the thing the model controls: a probability, a margin, an error or a distance. Drag across a plot, or use its slider, to move the dot along the curve, and read the loss and its gradient off the readout. Sliders under a card change the loss's own settings. Dashed and dotted curves are other losses, drawn for comparison.
01
What a loss function is
A loss scores one prediction against the truth. Training adjusts the model to make the average loss as small as possible.
training minimises (1/n) · Σi L(yi, ŷi)
A loss does two jobs. Its value says how wrong a prediction is, and its slope, the gradient, says which way to change the model to make it less wrong. A loss with no useful slope cannot be trained on, however well it describes the goal.
That is why a loss and a metric are not always the same. A metric, as in the evaluation series, is what you report. Accuracy is the natural metric for classification, but it is flat almost everywhere, so training uses a smooth stand-in such as cross-entropy. For regression the two often coincide: mean squared error (MSE) is both.
One result runs through the classification losses. With a sigmoid or a softmax in front, the gradient of cross-entropy with respect to the logit z is simply p − y: the predicted probability minus the truth. Part 3, on logistic regression, derives it.
02
Binary classification
Two classes, labelled y = 0 and y = 1, or y = −1 and y = +1 for the margin losses. The model outputs a score z; cross-entropy turns it into a probability p = σ(z) first, while the hinge family works on z directly through the margin y·z, which is positive when the prediction is on the right side.
Binary cross-entropy (BCE, log loss)
−log of the probability given to the truth
L = −[y log p + (1 − y) log(1 − p)] y = 1 leaves −log p; y = 0 leaves −log(1 − p) ∂L/∂z = p − y
Takes
a label y ∈ {0, 1} and a probability p = σ(z)
Gradient
p − y with respect to the logit z: small when right, largest when confidently wrong
Zero
only when p equals y exactly
Used for
any binary classifier, and logistic regression (part 3)
Watch out
−log p is infinite at p = 0, so one confident wrong answer can dominate. Pass the logit to BCEWithLogitsLoss rather than a sigmoid output to BCELoss: the same loss, computed stably
a predicted probability for every pixel, and the true mask
Measures
the overlap between prediction and mask, like an F1 score
Here
p on the object's pixels and q on the background, with the object a share f of the image
Used for
segmentation where the object is a small part of the image, such as a lesion in a scan. Drag f down: Dice keeps its full range while the mean cross-entropy, dominated by background pixels, barely changes
Watch out
unstable when the mask is empty, so it is often added to cross-entropy rather than used alone
Library
MONAI DiceLoss, segmentation_models_pytorch
The 0–1 loss, the fraction misclassified and drawn dotted on the hinge card, is what classification really cares about. Its gradient is zero everywhere it exists, so every loss above is a smooth stand-in for it. Log loss written with ±1 labels, log(1 + e−y·z), is binary cross-entropy under another name: PyTorch calls it SoftMarginLoss.
03
Multiclass classification
One of K classes is true. The model outputs a logit zk for each, and the softmax turns them into probabilities pk that sum to 1. The label is a one-hot vector y: 1 for the true class and 0 for every other.
Cross-entropy adds up −yk log pk over all K classes.
L = −Σk=1…K yk log pk
Because y is one-hot, every term but one is multiplied by 0. With three classes, the second one true, so y = (0, 1, 0), and the model predicting p = (0.2, 0.7, 0.1):
So for a one-hot label the sum is just L = −log py, minus the log of the probability given to the true class, and that is what the plots below show. With K = 2 the sum has two terms, y log p and (1 − y) log(1 − p), which is binary cross-entropy from section 02. Label smoothing is the exception: its y is not one-hot, so every term counts.
Cross-entropy (CE)
−Σ yk log pk
L = −Σk yk log pk, over all K classes y is one-hot, so only the true class is left: L = −log py ∂L/∂zk = pk − yk
Takes
a logit for each of K classes, and the true class as a one-hot vector y
Gradient
softmax − one-hot: p − y again, class by class
A uniform guess
1/K for every class scores log K
Used for
every multiclass classifier
Watch out
PyTorch's CrossEntropyLoss applies the softmax itself: pass logits, not probabilities, or it is applied twice
Library
PyTorch CrossEntropyLoss (logits), NLLLoss (log-probabilities); Keras categorical_crossentropy (one-hot labels) and sparse_categorical_crossentropy (integer labels); scikit-learn log_loss
Label-smoothed cross-entropy
aim for 1 − ε, not 1
target = 1 − ε + ε/K on the true class, ε/K on each other L = −Σk targetk log pk
Takes
logits, the true class, and a smoothing ε
Lowest at
py = 1 − ε + ε/K: the model is never pushed to certainty
Here
the other K − 1 classes share whatever py leaves
Used for
image classifiers and transformers, where ε = 0.1 is common. Predictions come out less over-confident and better calibrated
Watch out
it caps the confidence the model will reach, which hurts if you need probabilities near 1, and it can weaken knowledge distillation
Library
PyTorch CrossEntropyLoss(label_smoothing=0.1)
Weighted cross-entropy
wy × the cross-entropy
L = −wy log py
Takes
logits, the true class, and one weight per class
Effect
mistakes on a class with weight w cost w times as much
Typical weights
inversely proportional to how common each class is
Used for
imbalanced multiclass data, such as rare diagnoses among common ones
Watch out
PyTorch's mean divides by the sum of the weights in the batch, not by the number of examples, so the loss scale shifts from batch to batch
Library
PyTorch CrossEntropyLoss(weight=…)
Focal loss, multiclass
−(1 − py)γ log py
L = −αy (1 − py)γ log py, p from the softmax
Takes
logits, the true class, γ, and optionally a weight α per class
Same as
the binary focal loss, with pₜ the softmax probability of the true class
γ = 0
plain cross-entropy
Used for
multiclass detection and segmentation with many easy examples
Watch out
there is no PyTorch builtin; it is a few lines on top of log_softmax
Library
written by hand from F.log_softmax
Multiclass hinge
(1/K) Σ max(0, 1 − zy + zj)
L = (1/K) Σj≠y max(0, 1 − zy + zj)
Takes
a raw score for every class, not probabilities
Zero
once the true class's score beats every other by at least 1
Here
three classes; the other two scores are 0 and s
Used for
multiclass SVMs
Watch out
Crammer and Singer's version, in LinearSVC(multi_class="crammer_singer"), takes the largest term instead of the sum. Cross-entropy never reaches 0, so it keeps pushing the scores apart
a predicted distribution p and a target distribution q
Zero
only when p = q
vs cross-entropy
they differ by H(q), the target's entropy, which does not depend on the model, so the gradients are identical
Used for
knowledge distillation, matching a teacher model's soft outputs, and the regulariser in variational autoencoders
Watch out
PyTorch's KLDivLoss wants log-probabilities as input and probabilities as target, and reduction="batchmean" for the true KL
Library
PyTorch KLDivLoss
Multi-label is different. When several classes can be true at once, such as the tags on a photo, there are K separate yes-or-no questions. Use binary cross-entropy on each class, BCEWithLogitsLoss, not softmax with cross-entropy, which forces the classes to compete.
04
Regression
The model predicts a number ŷ, and most regression losses are a function of the error e = y − ŷ. A positive e means the prediction was too low.
Mean squared error (MSE, L2)
e²
L = mean (y − ŷ)² ∂L/∂ŷ = 2(ŷ − y)
Predicts
the mean of y
Gradient
proportional to the error, so big misses dominate
Units
squared units of y
Used for
the default for regression, and the least squares behind linear regression
Watch out
outliers dominate: one error of 10 costs as much as a hundred errors of 1
Library
PyTorch MSELoss; scikit-learn mean_squared_error
Mean absolute error (MAE, L1)
|e|
L = mean |y − ŷ| ∂L/∂ŷ = sign(ŷ − y)
Predicts
the median of y
Gradient
always ±1, whatever the size of the error
Units
the units of y
Used for
data with outliers, or when every unit of error costs the same
Watch out
the gradient stays full size right up to the answer, so training can bounce around the minimum; lower the learning rate as it goes
Library
PyTorch L1Loss; scikit-learn mean_absolute_error
Huber
e² near 0, |e| beyond δ
L = ½e² if |e| ≤ δ, else δ(|e| − ½δ) ∂L/∂ŷ = −e inside δ, −δ · sign(e) outside
Predicts
something between the mean and the median, set by δ
Gradient
like MSE near 0, capped at δ for large errors
δ
the error size where it switches from squared to linear
Used for
regression with some outliers, and bounding-box regression in object detection as Smooth L1
Watch out
δ is in the units of y, so it has to be chosen for each target
Library
PyTorch HuberLoss(delta); SmoothL1Loss(beta) is Huber divided by β; scikit-learn HuberRegressor
Log-cosh
log cosh e
L = log cosh(y − ŷ) ∂L/∂ŷ = −tanh(y − ŷ)
Near 0
about ½e², like MSE
Far out
about |e| − log 2, like MAE
Gradient
tanh of the error: smooth, and never above 1
Used for
a Huber-like loss with no δ to choose. It has a second derivative everywhere, which gradient boosting libraries use for custom objectives
Watch out
cosh overflows for errors beyond about 710; compute it as |e| + log(1 + e−2|e|) − log 2
Library
no builtin in PyTorch or scikit-learn: one line by hand
Quantile (pinball)
max(τe, (τ − 1)e)
L = max(τ·e, (τ − 1)·e), with e = y − ŷ ∂L/∂ŷ = −τ if e > 0, 1 − τ if e < 0
Predicts
the τ-th quantile of y: at τ = 0.9, a value that 90% of outcomes fall below
Shape
a tilted V: under-predicting costs τ per unit, over-predicting 1 − τ
τ = 0.5
half the MAE, so it predicts the median
Used for
prediction intervals, from one model at τ = 0.05 and one at τ = 0.95, and demand forecasting
Watch out
models trained separately for different τ can cross, giving a lower bound above the upper one
targets and predictions above −1, usually positive
Measures
relative error: predicting 20 for 10 costs about as much as 200 for 100
Here
the true value is y = 10
Used for
targets that span orders of magnitude, such as prices or counts
Watch out
an under-prediction costs more than an over-prediction of the same size, and it predicts the geometric mean of 1 + y, minus 1, so predictions sit below the average
Also seen: Gamma and Tweedie deviance, for positive, skewed targets such as insurance claim amounts, and mean absolute percentage error (MAPE) as a loss, which is rarely a good idea for the reasons in part 13 of the evaluation series.
05
Contrastive losses
These train an embedding: a model that maps each input to a vector. There is no class to predict. Training only needs to know which inputs belong together.
A positive pair is two inputs that should end up close: two photos of the same person, two random crops of the same image, or a photo and its own caption.
A negative pair is two inputs that should end up far apart: photos of two different people, crops of two different images, or a photo and another photo's caption.
Pairs come from labels, such as who is in each photo, or are built from the data itself: crop one image twice and you have a positive pair without labelling anything, which is how self-supervised learning works. In a batch, each input's positive is its partner, and the rest of the batch usually serves as its negatives. Closeness is measured as a distance d between the two vectors (0 when they are identical) or as a cosine similarity s (1 when they point the same way, −1 when they point opposite ways).
The pair loss takes one pair at a time and a label y: 1 for a positive pair, 0 for a negative pair. As with binary cross-entropy, one of its two terms is always 0, so a positive pair costs d² and a negative pair costs max(0, m − d)².
L = y · d² + (1 − y) · max(0, m − d)²
Contrastive loss (pairs)
pull positive pairs in, push negative pairs past m
L = y · d² + (1 − y) · max(0, m − d)² y = 1, a positive pair: L = d² y = 0, a negative pair: L = max(0, m − d)²
Takes
two embeddings, the distance d between them, and y: 1 for a positive pair, 0 for a negative pair
Positive pairs
cost d², so they are pulled together all the way to d = 0
Negative pairs
cost (m − d)² while closer than the margin m, and nothing once they are at least m apart
Used for
learning embeddings from pairs: face and signature verification, with Siamese networks (two copies of one network, sharing weights)
Watch out
conventions vary: some papers use y = 1 for a negative pair, put ½ in front, or use d instead of d². Most negative pairs are already far apart and contribute nothing, so training looks for hard negatives: negative pairs that are still close
Library
no core PyTorch class; pytorch-metric-learningContrastiveLoss, or a few lines by hand
InfoNCE takes one anchor at a time and compares it with N candidates: its positive and N − 1 negatives. With s+ the anchor's similarity to its positive, sj its similarity to negative j, and T a temperature:
L = −log [ es+/T / ( es+/T + Σj=1…N−1 esj/T ) ]
The numerator, es+/T, comes from the positive pair alone. It grows as the positive moves closer to the anchor.
The denominator is that same term plus one term for each negative: a total over all N candidates. Any negative that sits close to the anchor makes it bigger.
The fraction is a softmax over the N candidates: the probability the anchor gives to its own positive. It is near 1 when the positive is far more similar than every negative, and 1/N when the model cannot tell them apart. Its −log is cross-entropy with the positive as the true class, so the loss is 0 for a perfect match and log N for a blind guess. To lower it, training raises the numerator by pulling the positive in, and shrinks the negatives' share of the denominator by pushing them away. A small T exaggerates differences in similarity, so the closest negatives dominate the denominator.
For example, take an anchor whose positive has similarity 0.8, three negatives at 0.1, 0.3 and −0.3, and T = 0.5:
L = −log ( numerator / denominator ) numerator = es₊/T, the positive pair denominator = es₊/T + Σj esj/T, the positive and all N − 1 negatives
Takes
an anchor, its positive (s₊ is their cosine similarity) and N − 1 negatives (the rest of the batch)
It is
cross-entropy with N classes, where the right class is the positive. A blind guess scores log N, the dashed line
Here
every negative has similarity s₋ to the anchor
Used for
self-supervised and multimodal learning: SimCLR (a simple framework for contrastive learning) as NT-Xent, the normalised temperature-scaled cross-entropy; MoCo (momentum contrast); and CLIP (contrastive language–image pre-training), which matches each image to its own caption among the batch's
Watch out
the temperature matters a lot: small T sharpens the softmax and focuses on the hardest negatives. Bigger batches give more negatives, which is why these methods train with large ones
Library
F.cross_entropy(similarities / T, targets), as in the code below
Triplet loss sits between the two: max(0, d(a, p) − d(a, n) + m) takes an anchor a, a positive p and a negative n, and only asks that the negative be at least m further away than the positive. It is PyTorch's TripletMarginLoss, and it was used to train FaceNet.
06
What each loss predicts
For regression, the loss decides which summary of y the model learns to predict.
MSE → the mean · MAE → the median · pinball at τ → the τ-th quantile MSLE → the geometric mean of 1 + y, minus 1
Take a skewed sample of 4,001 values and find the single number that makes each loss smallest. For MSE it is 1.83, the sample's mean. For MAE it is 1.31, the median; for pinball at τ = 0.9 it is 3.68, the 90th percentile; and for MSLE it is 1.51. On symmetric data the first two agree; on skewed data, such as incomes or hospital stays, they can differ by a lot, and the loss you train with decides which one you get.
07
Which to use
problem
loss
why
Binary classification
binary cross-entropy
probabilities, and a p − y gradient
… with rare positives
weighted BCE or focal
stop the common class swamping the loss
Maximum-margin classifier
hinge
only points near the boundary matter
Segmenting small objects
Dice, often plus BCE
scores overlap, not pixel counts
Multiclass classification
cross-entropy
the standard; softmax probabilities
… over-confident models
label smoothing
caps the target below 1
… imbalanced classes
weighted CE or focal
rare classes count for more
Matching soft targets
KL divergence
compares whole distributions
Several labels at once
BCE per class
independent yes-or-no answers
Regression
MSE
predicts the mean
… with outliers
MAE or Huber
large errors do not dominate
… values spanning magnitudes
MSLE
relative error
… counts
Poisson
predicts a rate
… a prediction interval
quantile
predicts a chosen quantile
… with its own uncertainty
Gaussian NLL
predicts a mean and a variance
Learning embeddings from pairs
contrastive
distances, with a margin
Learning embeddings from batches
InfoNCE
one positive against many negatives
08
In code
Each loss is a line or two of NumPy. Below the definitions they are run on small examples, and the numbers in the comments match PyTorch and scikit-learn on the same inputs.
by_hand.py
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
def log_softmax(Z): Z = Z - Z.max(axis=1, keepdims=True); return Z - np.log(np.exp(Z).sum(axis=1, keepdims=True))
# ---- binary: y in {0, 1}, p = sigmoid(z) ----
def bce(y, p): return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
def weighted_bce(y, p, w): return -np.mean(w * y * np.log(p) + (1 - y) * np.log(1 - p))
def focal(y, p, gamma=2): pt = np.where(y == 1, p, 1 - p); return -np.mean((1 - pt) ** gamma * np.log(pt))
def hinge(y, z): return np.mean(np.maximum(0, 1 - (2 * y - 1) * z)) # labels mapped to -1, +1
# ---- multiclass: Z = logits, one row per example; y = the true class index ----
def cross_entropy(Z, y): # y = index of the true class
Y = np.eye(Z.shape[1])[y] # one-hot: 1 for the true class, 0 for the rest
return -np.mean(np.sum(Y * log_softmax(Z), axis=1)) # only the true class's term survives
def smoothed_ce(Z, y, eps=0.1):
K = Z.shape[1]; T = np.full(Z.shape, eps / K); T[np.arange(len(y)), y] += 1 - eps
return -np.mean(np.sum(T * log_softmax(Z), axis=1))
def kl(q, p): return np.sum(q * np.log(q / p))
# ---- regression: e = y - y_hat ----
def mse(y, yh): return np.mean((y - yh) ** 2)
def mae(y, yh): return np.mean(np.abs(y - yh))
def huber(y, yh, d=1.0): e = np.abs(y - yh); return np.mean(np.where(e <= d, 0.5 * e ** 2, d * (e - 0.5 * d)))
def log_cosh(y, yh): e = np.abs(y - yh); return np.mean(e + np.log1p(np.exp(-2 * e)) - np.log(2)) # no overflow
def pinball(y, yh, tau): e = y - yh; return np.mean(np.maximum(tau * e, (tau - 1) * e))
def gaussian_nll(y, mu, var): return np.mean(0.5 * (np.log(var) + (y - mu) ** 2 / var))
# ---- contrastive: embeddings, compared by distance or similarity ----
def contrastive(d, y, m=1.0): return np.mean(y * d ** 2 + (1 - y) * np.maximum(0, m - d) ** 2) # y = 1: positive pair
def info_nce(S, t=0.1): return cross_entropy(S / t, np.arange(len(S))) # row i: column i is the positive, the rest negatives
y = np.array([1, 0, 1, 0]); p = np.array([0.9, 0.2, 0.6, 0.4]); z = np.log(p / (1 - p))
bce(y, p) # 0.338
weighted_bce(y, p, w=4.0) # 0.800
focal(y, p, gamma=2) # 0.043
hinge(y, z) # 0.297
Z = np.array([[2.0, 0.5, -1.0], [0.1, 1.2, 0.3]]); c = np.array([0, 1])
cross_entropy(Z, c) # 0.397
smoothed_ce(Z, c, eps=0.1) # 0.506
kl(np.array([0.7, 0.3]), np.array([0.6, 0.4])) # 0.022
y_true = np.array([3.0, 5.0, 2.5, 7.0]); y_hat = np.array([2.5, 5.5, 2.0, 9.0])
mse(y_true, y_hat) # 1.188
mae(y_true, y_hat) # 0.875
huber(y_true, y_hat, d=1.0) # 0.469
log_cosh(y_true, y_hat) # 0.421
pinball(y_true, y_hat, 0.9) # 0.287
gaussian_nll(y_true, y_hat, var=np.full(4, 2.0)) # 0.643
contrastive(np.array([0.2, 0.9, 1.5, 0.4]), y=np.array([1, 1, 0, 0])) # 0.302, two positive pairs, two negative
S = np.array([[0.9, 0.1, -0.2], [0.0, 0.8, 0.3], [-0.1, 0.2, 0.7]]) # anchor i against candidate j; positives on the diagonal
info_nce(S, t=0.1) # 0.005
with_pytorch.py
import torch, torch.nn.functional as F
from torchvision.ops import sigmoid_focal_loss
from sklearn.metrics import hinge_loss, mean_pinball_loss
z = torch.logit(torch.tensor([0.9, 0.2, 0.6, 0.4])); y = torch.tensor([1., 0., 1., 0.])
F.binary_cross_entropy_with_logits(z, y) # 0.338, takes logits
F.binary_cross_entropy_with_logits(z, y, pos_weight=torch.tensor(4.)) # 0.800
sigmoid_focal_loss(z, y, alpha=-1, gamma=2, reduction="mean") # 0.043, alpha=-1: no class weight
hinge_loss(2 * y.numpy() - 1, z.numpy()) # 0.297, labels as -1, +1
Z = torch.tensor([[2.0, 0.5, -1.0], [0.1, 1.2, 0.3]]); c = torch.tensor([0, 1])
F.cross_entropy(Z, c) # 0.397, takes logits
F.cross_entropy(Z, c, label_smoothing=0.1) # 0.506
F.kl_div(torch.tensor([0.6, 0.4]).log(), torch.tensor([0.7, 0.3]), reduction="sum") # 0.022
y_true = torch.tensor([3.0, 5.0, 2.5, 7.0]); y_hat = torch.tensor([2.5, 5.5, 2.0, 9.0])
F.mse_loss(y_hat, y_true) # 1.188
F.l1_loss(y_hat, y_true) # 0.875
F.huber_loss(y_hat, y_true, delta=1.0) # 0.469
mean_pinball_loss(y_true, y_hat, alpha=0.9) # 0.287
F.gaussian_nll_loss(y_hat, y_true, var=torch.full((4,), 2.0)) # 0.643
S = torch.tensor([[0.9, 0.1, -0.2], [0.0, 0.8, 0.3], [-0.1, 0.2, 0.7]])
F.cross_entropy(S / 0.1, torch.arange(3)) # 0.005, InfoNCE is cross-entropy
torch.nn.TripletMarginLoss(margin=1.0) # anchor, positive, negative embeddings
Next, part 3:logistic regression, which puts the sigmoid from part 1 and binary cross-entropy together into a complete classifier, built from scratch.