Multiclass Logistic Regression

Machine learning from scratch · part 4Multiclass logistic regression

Logistic regression for more than two classes. One weighted sum per class, a softmax to turn them into probabilities that sum to 1, and the same p − y gradient as before.

This is part 3's logistic regression with one change: instead of one score and a sigmoid, the model computes one score per class and passes them through a softmax. Everything else carries over, so this page follows the same four steps and points back to part 3 wherever nothing changes. It ends by reading handwritten digits: MNIST, ten classes, 70,000 images.

01

What changes with more than two classes

Multiclass logistic regression predicts a probability for every class, and the probabilities sum to 1.

p = softmax(Wᵀx + b)

It is also called softmax regression or multinomial logistic regression. Use it whenever an example belongs to exactly one of K classes: which digit, which product category, which diagnosis. Binary logistic regression is the special case K = 2.

Input. A feature matrix X of shape (n, d), as in part 3. The labels y are now integers from 0 to K − 1.

Output. For each example, K probabilities, one per class, that sum to 1. The predicted class is the one with the highest probability, so there is no threshold to pick.

What it learns. A weight matrix W of shape (d, K), one column of weights per class, and K biases. For MNIST that is 784 × 10 weights plus 10 biases: 7,850 numbers.

02

Step 1: one score per class

Each class gets its own weighted sum of the features, exactly like part 3's linear score, computed K times:

zk = wk · x + bk   for k = 0, …, K − 1
in NumPy:  Z = X @ W + b   shape (n, K)

The predicted class is the one with the largest score. Two classes k and j tie where zk = zj, which is a straight line, so the plane splits into K regions with straight edges: the model is still linear. With three classes in two dimensions, each class's weights are an arrow, and you can drag them.

Three classes, three arrows

the largest score wins
  • class 0
  • class 1
  • class 2

zk = wk · x + bk  for k = 0, 1, 2
predicted class = the k with the largest zk

Drag
the tip of an arrow to change that class's weights
Each arrow
points toward where its class scores highest. Two regions meet where their scores tie: (wk − wj) · x + (bk − bj) = 0, a straight line
Train
runs gradient descent on all 300 points, from wherever the arrows are

Press train from zero and watch the arrows spread out, each toward its own cluster. They always add up to (0, 0): every gradient step changes them by amounts that sum to zero. That is a sign of something deeper. Only the differences between the weight vectors matter, so one class's weights could be fixed at 0 without losing anything. With K = 2 that leaves exactly one weight vector, which is part 3's model.

03

Step 2: softmax turns scores into probabilities

Part 3 squashed its one score with a sigmoid. With K scores, the softmax function from part 1 does the job: exponentiate each score, so it is positive, then divide by the total, so they sum to 1.

pk = ezk / ( ez0 + ez1 + … + ezK−1 )

For example, scores of 2, 1 and 0:

ez = (7.389, 2.718, 1.000), which sum to 11.107
p = (7.389 / 11.107, 2.718 / 11.107, 1.000 / 11.107) = (0.665, 0.245, 0.090)

The largest score always gets the largest probability, so taking the highest probability and taking the highest score give the same class. Adding the same number to every score leaves p unchanged, because the factor ec cancels from top and bottom. That is the arrows' freedom from section 02, and it also makes softmax safe to compute (section 07).

With two classes it is the sigmoid. Divide the top and bottom by ez1:

p1 = ez1 / (ez0 + ez1) = 1 / (1 + e−(z1 − z0)) = σ(z1 − z0)

That is part 3's sigmoid applied to the difference of the two scores, with weights w = w1 − w0. So part 3's model is this one with K = 2.

04

Step 3: measure the error with cross-entropy

The loss is the multiclass cross-entropy from part 2. The label becomes a one-hot vector y, with a 1 for the true class and 0 elsewhere, and since only one term survives, as part 2 works through, the loss is minus the log of the probability given to the true class:

loss = −mean( Σk yk log pk ) = −mean( log py )

With K = 2 this is part 3's log loss. It keeps the property that made log loss the right choice there: the gradient with respect to each score is simply pk − yk. Section 06 derives it. The card below shows it for one example with four classes.

One example, four classes

scores → softmax → loss → gradient
true class:

The loss
looks only at the true class: −log py. It is small when the true class gets most of the probability
The gradient
pk − yk for every class: negative for the true class, so its score goes up, and positive for each other class, by exactly the probability it took
Raise every score
by the same amount and nothing changes: softmax only sees differences
05

Step 4: gradient descent, in batches

The update is part 3's with matrices in place of vectors. Stack the one-hot labels into Y, of shape (n, K), and every class's gradient comes out at once:

P = softmax(X @ W + b)      # (n, K) probabilities
error = P - Y               # (n, K) prediction minus truth
dW = X.T @ error / n        # (d, K) one gradient column per class
db = error.mean(axis=0)     # (K,) one per class
W = W - lr * dW             # step downhill
b = b - lr * db

To remember, the same as part 3
dW = the input times the error, averaged:  Xᵀ(P − Y) / n, one column per class
db = the error, averaged:  the mean of each column of P − Y

The bias works like a weight on an input that is always 1, so its gradient is the error alone.

Part 3 used all n examples for every step. MNIST has 60,000 training images, so each full step would be slow, and the model would take only one step per pass over the data. Mini-batch gradient descent computes each step on a small random batch instead, 128 images here. One pass over the data, an epoch, then makes 469 steps instead of one. Each step's gradient is noisier, because it comes from 128 examples rather than all of them, but it costs about 1/469 as much, and the steps add up. The data is reshuffled every epoch, so the batches change.

The learning rate behaves as in part 3's training card: too small is slow, too large overshoots.

06

Where the gradient comes from

The chain rule runs as in part 3's derivation, with one new twist: every probability depends on every score, because they share the denominator. For one example, with S = Σj ezj:

pj = ezj / S
loss = −Σj yj log pj

1. The loss with respect to each probability.

∂loss/∂pj = −yj / pj

2. Each probability with respect to each score. Use the quotient rule. When j = k the top depends on zk too, which gives the sigmoid's slope from part 3. When j ≠ k only S does:

j = k:  ∂pk/∂zk = (ezk S − ezk ezk) / S² = pk(1 − pk)
j ≠ k:  ∂pj/∂zk = (0 · S − ezj ezk) / S² = −pj pk
together: ∂pj/∂zk = pj (δjk − pk), where δjk is 1 if j = k and 0 otherwise

3. Combine, summing over j, because zk reaches the loss through every pj. The pj cancels, and the last step uses Σj yj = 1, since exactly one yj is 1:

∂loss/∂zk = Σj (−yj / pj) · pj (δjk − pk)
          = −Σj yj δjk + pk Σj yj
          = −yk + pk
          = pk − yk

4. The weights. Only class k's score uses wk, and ∂zk/∂wk = x, as in part 3. So ∂loss/∂wk = (pk − yk) x and ∂loss/∂bk = pk − yk. Averaged over the n examples and stacked for all K classes, that is the code from section 05:

dW = Xᵀ (P − Y) / n     db = mean of the rows of (P − Y)

L2 regularization adds l2 · W to dW, as in part 3. And since every row of P − Y sums to 0, the K gradient columns always sum to zero: that is why the arrows in section 02 never drift together.

07

Practical details

Scale the pixels, don't standardize them. Part 3 standardized each feature. Here 67 of the 784 pixels are blank in every training image, so their standard deviation is 0 and standardizing would divide by zero. All pixels are already in the same unit, so dividing by 255 to get 0 to 1 is enough.

Compute softmax safely. ez overflows a 64-bit float once z passes about 709. Subtracting each row's largest score first leaves the probabilities unchanged, as section 03 showed, and keeps every exponent at 0 or below.

One-hot the labels. np.eye(K)[y] turns the labels 0 to K − 1 into rows of Y in one line.

Batch size and epochs. Small batches give cheap, noisy steps, and large batches give accurate, expensive ones. 32 to 512 is the usual range. Shuffle every epoch, so the model never sees the same batches in the same order.

Regularize. L2 works as in part 3, and so does scikit-learn's C: l2 = 1 / (C · n).

One-vs-rest is the alternative. Train K separate binary models from part 3, one class against all the others, and pick the most confident. It works, but each model is trained alone, so their probabilities do not sum to 1 until they are normalized. Softmax trains all K together. scikit-learn offers both, through OneVsRestClassifier.

Evaluate per class. Accuracy hides which classes fail. Look at the K × K confusion matrix and at recall per class, and summarize with macro or micro averages.

Time and space. One epoch computes X @ W and Xᵀ · E over all n examples: O(n · d · K), about 470 million multiply-adds for MNIST. Predicting one image takes O(d · K), 7,840 multiply-adds, and the model stores O(d · K) numbers.

08

The model, from scratch

The class mirrors part 3's, with W a matrix, softmax in place of the sigmoid, and the loop split into epochs and mini-batches.

softmax_regression.py

import numpy as np

class SoftmaxRegression:
    """Multiclass logistic regression: P = softmax(X @ W + b),
    trained by mini-batch gradient descent on cross-entropy."""

    def __init__(self, lr=0.5, epochs=10, batch_size=128, l2=0.0, seed=0):
        self.lr = lr                    # learning rate: the size of each step
        self.epochs = epochs            # passes over the whole training set
        self.batch_size = batch_size    # examples per gradient step
        self.l2 = l2                    # L2 regularization strength; 0 = none
        self.seed = seed                # fixes the order of the mini-batches
        self.W = None                   # one column of weights per class
        self.b = None                   # one bias per class
        self.losses = []                # the loss before and after each epoch

    def _softmax(self, Z):
        Z = Z - Z.max(axis=1, keepdims=True)    # same result, no overflow
        E = np.exp(Z)
        return E / E.sum(axis=1, keepdims=True) # each row now sums to 1

    def _loss(self, X, Y):
        P = np.clip(self._softmax(X @ self.W + self.b), 1e-15, 1)
        ce = -np.mean(np.sum(Y * np.log(P), axis=1))    # cross-entropy
        return ce + (self.l2 / 2) * np.sum(self.W ** 2) # plus the L2 penalty

    def fit(self, X, y):
        n, d = X.shape
        k = y.max() + 1                         # classes are 0, 1, ..., k-1
        Y = np.eye(k)[y]                        # one-hot: a 1 at column y[i]
        self.W = np.zeros((d, k))               # start at 0: every p is 1/k
        self.b = np.zeros(k)
        self.losses = [self._loss(X, Y)]        # before training: log k
        rng = np.random.default_rng(self.seed)
        for _ in range(self.epochs):
            order = rng.permutation(n)          # a new shuffle every epoch
            for start in range(0, n, self.batch_size):
                batch = order[start:start + self.batch_size]
                Xb, Yb = X[batch], Y[batch]
                # 1. predict: one score per class, then the softmax
                P = self._softmax(Xb @ self.W + self.b)
                # 2. the gradients; error is each example's share of them
                error = P - Yb
                dW = Xb.T @ error / len(batch) + self.l2 * self.W
                db = error.mean(axis=0)
                # 3. step downhill
                self.W -= self.lr * dW
                self.b -= self.lr * db
            self.losses.append(self._loss(X, Y))   # track progress
        return self

    def predict_proba(self, X):
        # one row per example, one column per class
        return self._softmax(X @ self.W + self.b)

    def predict(self, X):
        # the class with the highest probability, which is the highest score
        return np.argmax(X @ self.W + self.b, axis=1)
09

Run: MNIST

MNIST is 70,000 handwritten digits, 0 to 9, each a 28 × 28 grayscale image, with 60,000 for training and 10,000 for testing. Each image is flattened into 784 pixel features. The from-scratch class trains in seconds and reads 92.30% of the test digits correctly, with recall from 0.87 for 9s to 0.98 for 0s. Inside are the code, the output, the learned weights as images, test digits to try and the full confusion matrix.

showhide the code, output, weights, digits and confusion matrix

run_mnist.py

from sklearn.datasets import fetch_openml    # only to load the data

# 1. Load: 70,000 handwritten digits, 28 x 28 = 784 pixels each.
#    The first call downloads about 15 MB, then reads it from a cache.
X, y = fetch_openml("mnist_784", version=1, return_X_y=True, as_frame=False)
y = y.astype(int)                       # the labels arrive as strings

# 2. The standard split: the first 60,000 train, the last 10,000 test
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]

# 3. Scale the pixels from 0..255 to 0..1
X_train = X_train / 255.0
X_test = X_test / 255.0

# 4. Train: 10 epochs of 469 mini-batches each
model = SoftmaxRegression(lr=0.5, epochs=10, batch_size=128)
model.fit(X_train, y_train)

# 5. Evaluate on the test set, by hand
pred = model.predict(X_test)
cm = np.zeros((10, 10), dtype=int)
np.add.at(cm, (y_test, pred), 1)        # rows: truth; columns: prediction
recall = cm.diagonal() / cm.sum(axis=1)
off = cm - np.diag(cm.diagonal())       # the mistakes only
i, j = np.unravel_index(off.argmax(), off.shape)

print("loss before training:", round(model.losses[0], 4))
print("loss after 10 epochs:", round(model.losses[-1], 4))
print("test accuracy:", round(np.mean(pred == y_test), 4))
print("recall per digit:", recall.round(2))
print(f"most common mistake: {i} read as {j}, {off[i, j]} times")

output

loss before training: 2.3026                 # log 10: every p starts at 0.1
loss after 10 epochs: 0.2588
test accuracy: 0.923                         # 9230 of 10,000
recall per digit: [0.98 0.98 0.89 0.9  0.95 0.88 0.94 0.92 0.9  0.87]
most common mistake: 9 read as 4, 58 times   # see the matrix below

The loss starts at log 10 = 2.303, where every digit gets p = 0.1, and falls to 0.2588. Each class's 784 weights can be drawn as a 28 × 28 image. Some read like stencils: the 0 is a ring around a blue centre, where a 0 has no ink, and the 1 is a vertical stroke. Others are harder to read, because one linear template has to cover every way a digit is written.

What each class's weights look like

784 weights, as a 28 × 28 image
0
1
2
3
4
5
6
7
8
9
Orange
ink at this pixel raises the score of that digit
Blue
ink here lowers it: the model learned where each digit should be empty
Faint
a weight near 0, such as the border pixels that are always blank

A score is the sum of pixel × weight, so an image scores high for a digit when its ink lands on that digit's orange pixels and avoids its blue ones. Pick a test digit to see all ten probabilities:

Twenty test digits

dashed: the model gets it wrong

Computed live
from the 7,850 trained weights: 784 pixels times 10 columns, plus 10 biases, then a softmax
The wrong ones
are usually close calls, or digits written in a way the training set rarely shows

The most common mistake is 9 read as 4, 58 times. Every other one is in the matrix:

Every mistake on the test set

10 × 10 confusion matrix

Rows
the true digit; columns, the prediction, as in part 1 of the evaluation series
Diagonal
correct answers
Orange
mistakes; the darker, the more common
Hover
or tap a cell to read it

A linear model draws straight boundaries in the 784-dimensional pixel space, and about 8% of the test digits fall on the wrong side of them. Models that bend the boundaries, such as neural networks with the activation functions from part 1, get past 99%.

10

With scikit-learn

scikit-learn's LogisticRegression handles any number of classes and uses the softmax by default. On the same scaled data:

with_sklearn.py

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score

# defaults: softmax over all 10 classes, an L2 penalty with C=1.0, lbfgs.
# It warns that lbfgs stopped at its limit of 100 iterations.
sk = LogisticRegression()
sk.fit(X_train, y_train)                # the same scaled data as above
pred_sk = sk.predict(X_test)

print("test accuracy:", round(accuracy_score(y_test, pred_sk), 4))
print("macro F1:", round(f1_score(y_test, pred_sk, average="macro"), 4))
print("ours, macro F1:", round(f1_score(y_test, pred, average="macro"), 4))
print("same answer as ours:", round(np.mean(pred_sk == pred), 4))
print("weight shapes:", sk.coef_.shape, model.W.shape)

output

test accuracy: 0.9257                # ours: 0.9230
macro F1: 0.9247
ours, macro F1: 0.9219
same answer as ours: 0.9747
weight shapes: (10, 784) (784, 10)   # transposed: one row per class

scikit-learn adds an L2 penalty with C = 1.0 and solves with lbfgs (limited-memory Broyden–Fletcher–Goldfarb–Shanno). On MNIST it stops at its default limit of 100 iterations and raises a ConvergenceWarning. The accuracy is already there, and max_iter raises the limit. The two models agree on 97.5% of the test digits. scikit-learn stores the weights transposed, one row per class, and part 3 shows how to make the two match exactly.

11

In one paragraph

One linear score per class, a softmax to turn the scores into probabilities, and weights learned by minimizing cross-entropy with gradient descent.

The gradient for each score is pk − yk, so dW = Xᵀ(P − Y) / n, which is part 3's formula with one column per class: dW is the input times the error, averaged, and db is the error, averaged. With two classes it reduces to binary logistic regression. Training costs O(n · d · K) per epoch, and prediction costs O(d · K) per example.