Machine learning from scratch · part 4Multiclass logistic regression
Logistic regression for more than two classes. One weighted sum per class, a softmax to turn them into probabilities that sum to 1, and the same p − y gradient as before.
12 October 2026
18 min read
5 live plots
This is part 3's logistic regression with one change: instead of one score and a sigmoid, the model computes one score per class and passes them through a softmax. Everything else carries over, so this page follows the same four steps and points back to part 3 wherever nothing changes. It ends by reading handwritten digits: MNIST, ten classes, 70,000 images.
01
What changes with more than two classes
Multiclass logistic regression predicts a probability for every class, and the probabilities sum to 1.
p = softmax(Wᵀx + b)
It is also called softmax regression or multinomial logistic regression. Use it whenever an example belongs to exactly one of K classes: which digit, which product category, which diagnosis. Binary logistic regression is the special case K = 2.
Input. A feature matrix X of shape (n, d), as in part 3. The labels y are now integers from 0 to K − 1.
Output. For each example, K probabilities, one per class, that sum to 1. The predicted class is the one with the highest probability, so there is no threshold to pick.
What it learns. A weight matrix W of shape (d, K), one column of weights per class, and K biases. For MNIST that is 784 × 10 weights plus 10 biases: 7,850 numbers.
02
Step 1: one score per class
Each class gets its own weighted sum of the features, exactly like part 3's linear score, computed K times:
zk = wk · x + bk for k = 0, …, K − 1 in NumPy: Z = X @ W + b shape (n, K)
The predicted class is the one with the largest score. Two classes k and j tie where zk = zj, which is a straight line, so the plane splits into K regions with straight edges: the model is still linear. With three classes in two dimensions, each class's weights are an arrow, and you can drag them.
Three classes, three arrows
the largest score wins
class 0
class 1
class 2
zk = wk · x + bk for k = 0, 1, 2 predicted class = the k with the largest zk
Drag
the tip of an arrow to change that class's weights
Each arrow
points toward where its class scores highest. Two regions meet where their scores tie: (wk − wj) · x + (bk − bj) = 0, a straight line
Train
runs gradient descent on all 300 points, from wherever the arrows are
Press train from zero and watch the arrows spread out, each toward its own cluster. They always add up to (0, 0): every gradient step changes them by amounts that sum to zero. That is a sign of something deeper. Only the differences between the weight vectors matter, so one class's weights could be fixed at 0 without losing anything. With K = 2 that leaves exactly one weight vector, which is part 3's model.
03
Step 2: softmax turns scores into probabilities
Part 3 squashed its one score with a sigmoid. With K scores, the softmax function from part 1 does the job: exponentiate each score, so it is positive, then divide by the total, so they sum to 1.
pk = ezk / ( ez0 + ez1 + … + ezK−1 )
For example, scores of 2, 1 and 0:
ez = (7.389, 2.718, 1.000), which sum to 11.107 p = (7.389 / 11.107, 2.718 / 11.107, 1.000 / 11.107) = (0.665, 0.245, 0.090)
The largest score always gets the largest probability, so taking the highest probability and taking the highest score give the same class. Adding the same number to every score leaves p unchanged, because the factor ec cancels from top and bottom. That is the arrows' freedom from section 02, and it also makes softmax safe to compute (section 07).
With two classes it is the sigmoid. Divide the top and bottom by ez1:
That is part 3's sigmoid applied to the difference of the two scores, with weights w = w1 − w0. So part 3's model is this one with K = 2.
04
Step 3: measure the error with cross-entropy
The loss is the multiclass cross-entropy from part 2. The label becomes a one-hot vector y, with a 1 for the true class and 0 elsewhere, and since only one term survives, as part 2 works through, the loss is minus the log of the probability given to the true class:
loss = −mean( Σk yk log pk ) = −mean( log py )
With K = 2 this is part 3's log loss. It keeps the property that made log loss the right choice there: the gradient with respect to each score is simply pk − yk. Section 06 derives it. The card below shows it for one example with four classes.
One example, four classes
scores → softmax → loss → gradient
true class:
The loss
looks only at the true class: −log py. It is small when the true class gets most of the probability
The gradient
pk − yk for every class: negative for the true class, so its score goes up, and positive for each other class, by exactly the probability it took
Raise every score
by the same amount and nothing changes: softmax only sees differences
05
Step 4: gradient descent, in batches
The update is part 3's with matrices in place of vectors. Stack the one-hot labels into Y, of shape (n, K), and every class's gradient comes out at once:
P = softmax(X @ W + b) # (n, K) probabilities
error = P - Y # (n, K) prediction minus truth
dW = X.T @ error / n # (d, K) one gradient column per class
db = error.mean(axis=0) # (K,) one per class
W = W - lr * dW # step downhill
b = b - lr * db
To remember, the same as part 3 dW = the input times the error, averaged: Xᵀ(P − Y) / n, one column per class db = the error, averaged: the mean of each column of P − Y
The bias works like a weight on an input that is always 1, so its gradient is the error alone.
Part 3 used all n examples for every step. MNIST has 60,000 training images, so each full step would be slow, and the model would take only one step per pass over the data. Mini-batch gradient descent computes each step on a small random batch instead, 128 images here. One pass over the data, an epoch, then makes 469 steps instead of one. Each step's gradient is noisier, because it comes from 128 examples rather than all of them, but it costs about 1/469 as much, and the steps add up. The data is reshuffled every epoch, so the batches change.
The learning rate behaves as in part 3's training card: too small is slow, too large overshoots.
06
Where the gradient comes from
The chain rule runs as in part 3's derivation, with one new twist: every probability depends on every score, because they share the denominator. For one example, with S = Σj ezj:
pj = ezj / S loss = −Σj yj log pj
1. The loss with respect to each probability.
∂loss/∂pj = −yj / pj
2. Each probability with respect to each score. Use the quotient rule. When j = k the top depends on zk too, which gives the sigmoid's slope from part 3. When j ≠ k only S does:
j = k: ∂pk/∂zk = (ezk S − ezk ezk) / S² = pk(1 − pk) j ≠ k: ∂pj/∂zk = (0 · S − ezj ezk) / S² = −pj pk together: ∂pj/∂zk = pj (δjk − pk), where δjk is 1 if j = k and 0 otherwise
3. Combine, summing over j, because zk reaches the loss through every pj. The pj cancels, and the last step uses Σj yj = 1, since exactly one yj is 1:
4. The weights. Only class k's score uses wk, and ∂zk/∂wk = x, as in part 3. So ∂loss/∂wk = (pk − yk) x and ∂loss/∂bk = pk − yk. Averaged over the n examples and stacked for all K classes, that is the code from section 05:
dW = Xᵀ (P − Y) / n db = mean of the rows of (P − Y)
L2 regularization adds l2 · W to dW, as in part 3. And since every row of P − Y sums to 0, the K gradient columns always sum to zero: that is why the arrows in section 02 never drift together.
07
Practical details
Scale the pixels, don't standardize them.Part 3 standardized each feature. Here 67 of the 784 pixels are blank in every training image, so their standard deviation is 0 and standardizing would divide by zero. All pixels are already in the same unit, so dividing by 255 to get 0 to 1 is enough.
Compute softmax safely. ez overflows a 64-bit float once z passes about 709. Subtracting each row's largest score first leaves the probabilities unchanged, as section 03 showed, and keeps every exponent at 0 or below.
One-hot the labels.np.eye(K)[y] turns the labels 0 to K − 1 into rows of Y in one line.
Batch size and epochs. Small batches give cheap, noisy steps, and large batches give accurate, expensive ones. 32 to 512 is the usual range. Shuffle every epoch, so the model never sees the same batches in the same order.
Regularize. L2 works as in part 3, and so does scikit-learn's C: l2 = 1 / (C · n).
One-vs-rest is the alternative. Train K separate binary models from part 3, one class against all the others, and pick the most confident. It works, but each model is trained alone, so their probabilities do not sum to 1 until they are normalized. Softmax trains all K together. scikit-learn offers both, through OneVsRestClassifier.
Time and space. One epoch computes X @ W and Xᵀ · E over all n examples: O(n · d · K), about 470 million multiply-adds for MNIST. Predicting one image takes O(d · K), 7,840 multiply-adds, and the model stores O(d · K) numbers.
08
The model, from scratch
The class mirrors part 3's, with W a matrix, softmax in place of the sigmoid, and the loop split into epochs and mini-batches.
softmax_regression.py
import numpy as np
class SoftmaxRegression:
"""Multiclass logistic regression: P = softmax(X @ W + b),
trained by mini-batch gradient descent on cross-entropy."""
def __init__(self, lr=0.5, epochs=10, batch_size=128, l2=0.0, seed=0):
self.lr = lr # learning rate: the size of each step
self.epochs = epochs # passes over the whole training set
self.batch_size = batch_size # examples per gradient step
self.l2 = l2 # L2 regularization strength; 0 = none
self.seed = seed # fixes the order of the mini-batches
self.W = None # one column of weights per class
self.b = None # one bias per class
self.losses = [] # the loss before and after each epoch
def _softmax(self, Z):
Z = Z - Z.max(axis=1, keepdims=True) # same result, no overflow
E = np.exp(Z)
return E / E.sum(axis=1, keepdims=True) # each row now sums to 1
def _loss(self, X, Y):
P = np.clip(self._softmax(X @ self.W + self.b), 1e-15, 1)
ce = -np.mean(np.sum(Y * np.log(P), axis=1)) # cross-entropy
return ce + (self.l2 / 2) * np.sum(self.W ** 2) # plus the L2 penalty
def fit(self, X, y):
n, d = X.shape
k = y.max() + 1 # classes are 0, 1, ..., k-1
Y = np.eye(k)[y] # one-hot: a 1 at column y[i]
self.W = np.zeros((d, k)) # start at 0: every p is 1/k
self.b = np.zeros(k)
self.losses = [self._loss(X, Y)] # before training: log k
rng = np.random.default_rng(self.seed)
for _ in range(self.epochs):
order = rng.permutation(n) # a new shuffle every epoch
for start in range(0, n, self.batch_size):
batch = order[start:start + self.batch_size]
Xb, Yb = X[batch], Y[batch]
# 1. predict: one score per class, then the softmax
P = self._softmax(Xb @ self.W + self.b)
# 2. the gradients; error is each example's share of them
error = P - Yb
dW = Xb.T @ error / len(batch) + self.l2 * self.W
db = error.mean(axis=0)
# 3. step downhill
self.W -= self.lr * dW
self.b -= self.lr * db
self.losses.append(self._loss(X, Y)) # track progress
return self
def predict_proba(self, X):
# one row per example, one column per class
return self._softmax(X @ self.W + self.b)
def predict(self, X):
# the class with the highest probability, which is the highest score
return np.argmax(X @ self.W + self.b, axis=1)
09
Run: MNIST
MNIST is 70,000 handwritten digits, 0 to 9, each a 28 × 28 grayscale image, with 60,000 for training and 10,000 for testing. Each image is flattened into 784 pixel features. The from-scratch class trains in seconds and reads 92.30% of the test digits correctly, with recall from 0.87 for 9s to 0.98 for 0s. Inside are the code, the output, the learned weights as images, test digits to try and the full confusion matrix.
showhide the code, output, weights, digits and confusion matrix
run_mnist.py
from sklearn.datasets import fetch_openml # only to load the data
# 1. Load: 70,000 handwritten digits, 28 x 28 = 784 pixels each.
# The first call downloads about 15 MB, then reads it from a cache.
X, y = fetch_openml("mnist_784", version=1, return_X_y=True, as_frame=False)
y = y.astype(int) # the labels arrive as strings
# 2. The standard split: the first 60,000 train, the last 10,000 test
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]
# 3. Scale the pixels from 0..255 to 0..1
X_train = X_train / 255.0
X_test = X_test / 255.0
# 4. Train: 10 epochs of 469 mini-batches each
model = SoftmaxRegression(lr=0.5, epochs=10, batch_size=128)
model.fit(X_train, y_train)
# 5. Evaluate on the test set, by hand
pred = model.predict(X_test)
cm = np.zeros((10, 10), dtype=int)
np.add.at(cm, (y_test, pred), 1) # rows: truth; columns: prediction
recall = cm.diagonal() / cm.sum(axis=1)
off = cm - np.diag(cm.diagonal()) # the mistakes only
i, j = np.unravel_index(off.argmax(), off.shape)
print("loss before training:", round(model.losses[0], 4))
print("loss after 10 epochs:", round(model.losses[-1], 4))
print("test accuracy:", round(np.mean(pred == y_test), 4))
print("recall per digit:", recall.round(2))
print(f"most common mistake: {i} read as {j}, {off[i, j]} times")
output
loss before training: 2.3026 # log 10: every p starts at 0.1
loss after 10 epochs: 0.2588
test accuracy: 0.923 # 9230 of 10,000
recall per digit: [0.98 0.98 0.89 0.9 0.95 0.88 0.94 0.92 0.9 0.87]
most common mistake: 9 read as 4, 58 times # see the matrix below
The loss starts at log 10 = 2.303, where every digit gets p = 0.1, and falls to 0.2588. Each class's 784 weights can be drawn as a 28 × 28 image. Some read like stencils: the 0 is a ring around a blue centre, where a 0 has no ink, and the 1 is a vertical stroke. Others are harder to read, because one linear template has to cover every way a digit is written.
What each class's weights look like
784 weights, as a 28 × 28 image
0123456789
Orange
ink at this pixel raises the score of that digit
Blue
ink here lowers it: the model learned where each digit should be empty
Faint
a weight near 0, such as the border pixels that are always blank
A score is the sum of pixel × weight, so an image scores high for a digit when its ink lands on that digit's orange pixels and avoids its blue ones. Pick a test digit to see all ten probabilities:
Twenty test digits
dashed: the model gets it wrong
Computed live
from the 7,850 trained weights: 784 pixels times 10 columns, plus 10 biases, then a softmax
The wrong ones
are usually close calls, or digits written in a way the training set rarely shows
The most common mistake is 9 read as 4, 58 times. Every other one is in the matrix:
A linear model draws straight boundaries in the 784-dimensional pixel space, and about 8% of the test digits fall on the wrong side of them. Models that bend the boundaries, such as neural networks with the activation functions from part 1, get past 99%.
10
With scikit-learn
scikit-learn's LogisticRegression handles any number of classes and uses the softmax by default. On the same scaled data:
with_sklearn.py
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
# defaults: softmax over all 10 classes, an L2 penalty with C=1.0, lbfgs.
# It warns that lbfgs stopped at its limit of 100 iterations.
sk = LogisticRegression()
sk.fit(X_train, y_train) # the same scaled data as above
pred_sk = sk.predict(X_test)
print("test accuracy:", round(accuracy_score(y_test, pred_sk), 4))
print("macro F1:", round(f1_score(y_test, pred_sk, average="macro"), 4))
print("ours, macro F1:", round(f1_score(y_test, pred, average="macro"), 4))
print("same answer as ours:", round(np.mean(pred_sk == pred), 4))
print("weight shapes:", sk.coef_.shape, model.W.shape)
output
test accuracy: 0.9257 # ours: 0.9230
macro F1: 0.9247
ours, macro F1: 0.9219
same answer as ours: 0.9747
weight shapes: (10, 784) (784, 10) # transposed: one row per class
scikit-learn adds an L2 penalty with C = 1.0 and solves with lbfgs (limited-memory Broyden–Fletcher–Goldfarb–Shanno). On MNIST it stops at its default limit of 100 iterations and raises a ConvergenceWarning. The accuracy is already there, and max_iter raises the limit. The two models agree on 97.5% of the test digits. scikit-learn stores the weights transposed, one row per class, and part 3 shows how to make the two match exactly.
11
In one paragraph
One linear score per class, a softmax to turn the scores into probabilities, and weights learned by minimizing cross-entropy with gradient descent.
The gradient for each score is pk − yk, so dW = Xᵀ(P − Y) / n, which is part 3's formula with one column per class: dW is the input times the error, averaged, and db is the error, averaged. With two classes it reduces to binary logistic regression. Training costs O(n · d · K) per epoch, and prediction costs O(d · K) per example.