Logistic Regression

Machine learning from scratch · part 3Logistic regression

A model for yes-or-no questions. It predicts the probability that an example belongs to class 1, from a weighted sum of its features passed through a sigmoid. Despite the name, it classifies.

Logistic regression is built from two pieces covered earlier in the series: the sigmoid activation from part 1 and the binary cross-entropy loss from part 2. Both are written out again below where they are needed. The page builds the model in four steps, derives its gradient, writes it in NumPy from scratch, runs it on two datasets and checks it against scikit-learn.

01

What it does

Logistic regression predicts the probability that an example belongs to class 1.

p = σ(w · x + b)

Class 1 is whichever outcome you want to detect: fraud or not, default or not, malignant or benign. Despite the name, the model is used for classification, not regression; the name comes from the logistic function, another name for the sigmoid. It is widely used in banking and medicine because it is fast, simple, and its weights are easy to explain.

Input. A feature matrix X of shape (n, d): n examples with d features each, such as income and age. For training, also the labels y: n values, each 0 or 1.

Output. For each example, a probability p between 0 and 1 that it is class 1. A threshold, usually 0.5, turns it into a class.

What it learns. A weight vector w, one weight per feature, and a bias b.

02

Step 1: a linear score

Combine the features into one number with a weighted sum, as in linear regression:

z = w₁x₁ + w₂x₂ + … + wdxd + b
in NumPy:  z = X @ w + b

z can be any number, from −∞ to ∞, so it is not yet a probability. But it already decides the class: the model will say class 1 when z > 0 and class 0 when z < 0. The points where z = 0 form the decision boundary. With two features that is a straight line, and in more dimensions a flat plane, so logistic regression separates classes linearly.

The plot below holds the 320 training points of the first dataset in section 09: two overlapping clusters with two features each. Move the weights and watch the boundary turn and slide.

The boundary, by hand

z = w₁x₁ + w₂x₂ + b
  • class 1
  • class 0
  • boundary, p = 0.5
  • p = 0.1 and 0.9

Shading
orange where p is above 0.5, grey below; the deeper the colour, the more certain
w
the arrow: it points across the boundary toward class 1. Doubling it keeps the line but sharpens the change from 0 to 1
b
slides the line without turning it

Press the trained weights: the boundary runs diagonally between the two clusters, and a few points of each class still sit on the wrong side. The clusters overlap, so no straight line can separate them perfectly.

03

Step 2: squash it into a probability

The sigmoid function, from part 1, maps any z into the range 0 to 1:

σ(z) = 1 / (1 + e−z)

z = −5 → 0.007   almost surely class 0
z =  0 → 0.5     unsure
z =  5 → 0.993   almost surely class 1

The sigmoid is exactly 0.5 at z = 0, so the boundary where p = 0.5 is the same line where z = 0. Run the sigmoid backwards and z turns out to be the log-odds: z = log(p / (1 − p)). The model is linear in the log-odds, and that is what makes its weights readable: each weight is the change in log-odds per unit of its feature.

Every point, through the sigmoid

(−∞, ∞) → (0, 1)
  • class 1
  • class 0

p = σ(z) = 1 / (1 + e−z)
z = log(p / (1 − p))

The dots
the 320 training points, each at its own z for the weights set in the card above. Change them there and the dots move here
Good weights
put the orange dots high on the right and the hollow ones low on the left
04

Step 3: measure the error with log loss

Training needs one number that says how wrong the predictions are. Logistic regression uses log loss, also called binary cross-entropy, from part 2:

loss = −mean( y · log(p) + (1 − y) · log(1 − p) )

If the true label is 1, the second term is multiplied by 0 and only −log(p) counts: small when p is near 1, huge when p is near 0. If the label is 0, only −log(1 − p) counts. So confident wrong answers are punished heavily.

Why not mean squared error (MSE)? It works for linear regression, but put a sigmoid in front of it and two things go wrong. The loss is no longer convex, so it can have flat stretches and local minima where gradient descent stalls. And its gradient vanishes on confidently wrong examples. Log loss is convex in w and b, a single bowl, so gradient descent reliably finds the best weights. The card plots both losses for one example whose true class is 1.

Log loss against squared error

one example, true class 1
  • log loss, −log p
  • squared error, (1 − p)²
  • its gradient, p − 1
  • its gradient, −2p(1 − p)²

Log loss
keeps growing as the answer gets more confidently wrong, and its gradient stays near −1: the worst mistakes get the strongest correction
Squared error
can never exceed 1, and its gradient fades to 0 on the far left: the model stops learning from exactly the examples it gets most wrong
05

Step 4: learn the weights with gradient descent

Start with every weight at 0. Then repeat: compute the predictions, and move each weight a small step in the direction that lowers the loss. The gradient of log loss simplifies nicely; section 06 derives it.

error = p - y               # prediction minus truth
dw = X.T @ error / n        # gradient for the weights
db = np.mean(error)         # gradient for the bias
w = w - lr * dw             # step downhill
b = b - lr * db

To remember
dw = the input times the error, averaged:  Xᵀ(p − y) / n
db = the error, averaged:  mean(p − y)

The bias works like a weight on an input that is always 1, so its gradient is the error alone.

The learning rate lr sets the step size, and the number of iterations sets how long training runs. The card runs exactly this loop on the 320 points.

Gradient descent, step by step

w ← w − lr · dw
  • class 1
  • class 0
  • boundary, p = 0.5
  • p = 0.1 and 0.9

    Start
    every weight at 0, so every p is 0.5 and the loss is log 2 = 0.693
    Each step
    the gradient is computed on all 320 points, then w and b move against it
    lr
    changing it starts a new run, and the last run stays on the loss plot as a dashed line

    At lr = 0.1, 1000 steps land on w = (2.923, 2.533) and b ≈ 0, with a loss of 0.1777: the same numbers the code prints in section 09. Now try other rates. At 0.01 the loss is still falling after 1000 steps: too slow. At 10 it is done within five steps. At 100 each step overshoots, and the loss jumps up and down before it settles: too big.

    06

    Where the gradient comes from

    Gradient descent needs the gradient of the loss with respect to every weight. Here it is derived for one example, with its linear score z, its predicted probability p and its cross-entropy loss, using the natural log:

    z = w · x + b
    p = σ(z) = 1 / (1 + e−z)
    loss = −[ y log p + (1 − y) log(1 − p) ]

    The loss depends on w only through p, and p only through z, so the chain rule splits the gradient into three factors:

    ∂loss/∂wj = ∂loss/∂p · ∂p/∂z · ∂z/∂wj
    ∂loss/∂b  = ∂loss/∂p · ∂p/∂z · ∂z/∂b

    1. The loss with respect to p. Use d/dp log(p) = 1/p and d/dp log(1 − p) = −1/(1 − p), then put both fractions over the common denominator p(1 − p). The yp terms cancel:

    ∂loss/∂p = −[ y · (1/p) + (1 − y) · (−1/(1 − p)) ]
             = −y/p + (1 − y)/(1 − p)
             = [ −y(1 − p) + (1 − y)p ] / [ p(1 − p) ]
             = [ −y + yp + p − yp ] / [ p(1 − p) ]
             = (p − y) / [ p(1 − p) ]

    2. The sigmoid with respect to z. Write p = (1 + e−z)−1 and differentiate. The last step uses e−z / (1 + e−z) = 1 − p. This is the sigmoid's slope, plotted on its card in part 1:

    ∂p/∂z = −1 · (1 + e−z)−2 · (−e−z)
           = e−z / (1 + e−z)²
           = [ 1 / (1 + e−z) ] · [ e−z / (1 + e−z) ]
           = p (1 − p)

    3. Combine. The p(1 − p) cancels, leaving the p − y that part 2 promised:

    ∂loss/∂z = ∂loss/∂p · ∂p/∂z
             = (p − y) / [ p(1 − p) ] · p(1 − p)
             = p − y

    4. The score with respect to each weight. Since z = w₁x₁ + w₂x₂ + … + b, each weight's derivative is its own feature:

    ∂z/∂wj = xj       ∂z/∂b = 1

    5. The gradients for one example:

    ∂loss/∂wj = (p − y) · xj
    ∂loss/∂b  = p − y

    6. Average over all n examples. The total loss is an average over the n examples, so the gradients are averages too. Vectorized, this is exactly the code:

    dw = (1/n) Σi (pi − yi) xi  = Xᵀ (p − y) / n
    db = (1/n) Σi (pi − yi)     = mean(p − y)

    The update is then w = w − lr · dw and b = b − lr · db.

    With L2 regularization, a penalty on the squared weights, the loss gets an extra term (l2 / 2) · Σ wj². Its derivative adds l2 · w to dw: that is the + self.l2 * self.w in the code.

    So error = p − y is not the loss. It is each example's contribution to the gradient. The loss itself, with the logs, is computed separately only to track progress: the weights would train exactly the same without it.

    07

    Practical details

    Scale the features. Subtract the mean and divide by the standard deviation, using the training set's statistics only, so the test set stays unseen. Without it, a feature measured in thousands dominates one measured in fractions, and one learning rate cannot suit both, so training converges slowly.

    Regularize. L2 regularization adds l2 · w to the gradient, which keeps the weights small and reduces overfitting. scikit-learn sets the same penalty with C, which works the other way round: l2 = 1 / (C · n), so a smaller C means a stronger penalty. Section 11 shows the two giving the same model.

    Clip z. e−z overflows a 64-bit float once −z passes about 709, so the code clips z to ±500 first. The sigmoid of ±500 is 0 or 1 to every digit a float can hold, so nothing is lost.

    Read the weights. A positive weight pushes toward class 1 and a negative one toward class 0. Each weight is the change in log-odds per unit of its feature, so ew is the factor the odds are multiplied by. With standardized features, the unit is one standard deviation.

    Choose the threshold. 0.5 is only a default. In fraud detection, a lower threshold catches more fraud, a higher recall, at the cost of more false alarms, a lower precision. Section 10 lets you move it on real data.

    Time and space. One training step computes X @ w and Xᵀ · error, each O(n · d), so k steps cost O(k · n · d). Predicting one example costs O(d), and the model stores only its d weights and a bias: O(d) space.

    08

    The model, from scratch

    The whole model is one class in NumPy. fit is the loop from section 05, and the comments point back to the steps above.

    logistic_regression.py

    import numpy as np
    
    class LogisticRegression:
        """p = sigmoid(X @ w + b), trained by gradient descent on log loss."""
    
        def __init__(self, lr=0.1, n_iters=1000, l2=0.0):
            self.lr = lr              # learning rate: the size of each step
            self.n_iters = n_iters    # how many steps to take
            self.l2 = l2              # L2 regularization strength; 0 = none
            self.w = None             # one weight per feature, set by fit
            self.b = 0.0              # the bias
            self.losses = []          # the loss at every step, to track training
    
        def _sigmoid(self, z):
            z = np.clip(z, -500, 500)       # keep exp(-z) from overflowing
            return 1 / (1 + np.exp(-z))     # any number -> between 0 and 1
    
        def _loss(self, y, p):
            eps = 1e-15
            p = np.clip(p, eps, 1 - eps)    # log(0) would be minus infinity
            # binary cross-entropy, plus the L2 penalty
            bce = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
            return bce + (self.l2 / 2) * np.sum(self.w ** 2)
    
        def fit(self, X, y):
            n_samples, n_features = X.shape
            self.w = np.zeros(n_features)   # start at 0: every p starts at 0.5
            self.b = 0.0
            self.losses = []
            for _ in range(self.n_iters):
                # 1. predict: the linear score, then the sigmoid
                p = self._sigmoid(X @ self.w + self.b)
                # 2. record the loss, only to track progress
                self.losses.append(self._loss(y, p))
                # 3. the gradients; error is each example's share of them
                error = p - y
                dw = X.T @ error / n_samples + self.l2 * self.w
                db = np.mean(error)
                # 4. step downhill
                self.w -= self.lr * dw
                self.b -= self.lr * db
            return self
    
        def predict_proba(self, X):
            # the probability of class 1
            return self._sigmoid(X @ self.w + self.b)
    
        def predict(self, X, threshold=0.5):
            # probability -> class 0 or 1
            return (self.predict_proba(X) >= threshold).astype(int)
    09

    Run 1: two clusters

    Two overlapping clusters of 200 points each, made with NumPy alone: the 320 training points plotted in sections 02 to 05. The model ends with a loss of 0.1777 and a test accuracy of 0.8875, 71 of 80.

    showhide the code and its output

    The code makes the two clusters, splits them 80/20, scales them, trains and evaluates.

    run_clusters.py

    # 1. Make data: 2 features, 2 classes
    np.random.seed(42)
    X0 = np.random.randn(200, 2) - 1          # class 0, centred at (-1, -1)
    X1 = np.random.randn(200, 2) + 1          # class 1, centred at (1, 1)
    X = np.vstack([X0, X1])
    y = np.array([0] * 200 + [1] * 200)
    
    # 2. Shuffle and split 80% train, 20% test
    idx = np.random.permutation(len(X))
    X, y = X[idx], y[idx]
    split = int(0.8 * len(X))
    X_train, X_test = X[:split], X[split:]
    y_train, y_test = y[:split], y[split:]
    
    # 3. Standardize with training statistics only
    mean, std = X_train.mean(axis=0), X_train.std(axis=0)
    X_train = (X_train - mean) / std
    X_test = (X_test - mean) / std
    
    # 4. Train
    model = LogisticRegression(lr=0.1, n_iters=1000)
    model.fit(X_train, y_train)
    
    # 5. Evaluate
    print("first loss:", round(model.losses[0], 4))
    print("final loss:", round(model.losses[-1], 4))
    print("weights:", model.w.round(3), "bias:", round(model.b, 3))
    print("test accuracy:", np.mean(model.predict(X_test) == y_test))
    print("probabilities:", model.predict_proba(X_test[:3]).round(3))

    output

    first loss: 0.6931                   # log 2: every p starts at 0.5
    final loss: 0.1777                   # the model learned
    weights: [2.923 2.533] bias: -0.0    # both push toward class 1
    test accuracy: 0.8875                # 71 of 80: the two clusters overlap
    probabilities: [0.324 0.435 0.995]   # for the first three test points

    A falling loss confirms that training works: it starts at log 2 = 0.693, where every prediction is 0.5, and ends at 0.1777. Both weights are positive because class 1 sits up and to the right. The bias prints as −0.0, a tiny negative number rounded: the classes are balanced, so the boundary passes almost through the centre. The accuracy is 71 of 80 rather than all of them because the clusters overlap: no straight line can separate them perfectly.

    10

    Run 2: breast cancer

    A standard real dataset: 569 breast tumours, 212 malignant and 357 benign, with 30 measurements each. The same class, unchanged, reaches a test accuracy of 0.9737 and misses 2 of the 47 cancers in the test set. Inside are the code and its output, a threshold you can move, and the weights the model learned.

    showhide the code, output, threshold and weights

    The dataset ships with scikit-learn. Each tumour is described by 30 measurements of the cell nuclei in a digitized image of a needle biopsy: the mean, the standard error and the worst (largest) value of ten properties such as radius, texture and concavity. scikit-learn labels malignant as 0; the code flips it, so that class 1 is the cancer the model should catch.

    The model is the same class, unchanged, with the same settings. Evaluation is done by hand first, as in the evaluation metrics series: the confusion matrix, precision, recall and the ROC AUC, the area under the receiver operating characteristic curve.

    run_breast_cancer.py

    from sklearn.datasets import load_breast_cancer   # only to load the data
    
    # 1. Load: 569 tumours, 30 measurements each
    data = load_breast_cancer()
    X = data.data
    y = 1 - data.target   # malignant is 0 in scikit-learn; flip so 1 = malignant
    
    # 2. Shuffle and split 80% train, 20% test
    np.random.seed(42)
    idx = np.random.permutation(len(X))
    X, y = X[idx], y[idx]
    split = int(0.8 * len(X))
    X_train, X_test = X[:split], X[split:]
    y_train, y_test = y[:split], y[split:]
    
    # 3. Standardize with training statistics only
    mean, std = X_train.mean(axis=0), X_train.std(axis=0)
    X_train = (X_train - mean) / std
    X_test = (X_test - mean) / std
    
    # 4. Train, with the same settings as before
    model = LogisticRegression(lr=0.1, n_iters=1000)
    model.fit(X_train, y_train)
    
    # 5. Evaluate on the test set, by hand
    p = model.predict_proba(X_test)
    pred = model.predict(X_test)
    tp = int(np.sum((pred == 1) & (y_test == 1)))    # malignant, flagged
    fn = int(np.sum((pred == 0) & (y_test == 1)))    # malignant, missed
    fp = int(np.sum((pred == 1) & (y_test == 0)))    # benign, flagged
    tn = int(np.sum((pred == 0) & (y_test == 0)))    # benign, cleared
    # ROC AUC: the chance that a malignant tumour outscores a benign one
    auc = np.mean(p[y_test == 1][:, None] > p[y_test == 0][None, :])
    
    print("first loss:", round(model.losses[0], 4))
    print("final loss:", round(model.losses[-1], 4))
    print("test accuracy:", round(np.mean(pred == y_test), 4))
    print("confusion matrix:", [[tn, fp], [fn, tp]])
    print("precision:", round(tp / (tp + fp), 4))
    print("recall:", round(tp / (tp + fn), 4))
    print("ROC AUC:", round(auc, 4))
    top = np.argsort(model.w)[::-1][:3]
    print("strongest push to malignant:", ", ".join(data.feature_names[top]))

    output

    first loss: 0.6931                     # log 2 again
    final loss: 0.0523
    test accuracy: 0.9737                  # 111 of 114
    confusion matrix: [[66, 1], [2, 45]]   # rows: truth; columns: prediction
    precision: 0.9783                      # 45 of the 46 flagged are malignant
    recall: 0.9574                         # 45 of the 47 malignant are caught
    ROC AUC: 0.9933
    strongest push to malignant: worst texture, radius error, worst radius

    The 2 malignant tumours scored below 0.5 are the costly mistake: a missed cancer. The threshold is the lever. Lower it and those cases get flagged, at the price of more benign tumours sent for a second look. The card shows all 114 test tumours, each at the probability the model gave it.

    Choosing the threshold

    114 test tumours
    • ROC curve
    • random guessing
    pred. 0
    benign
    pred. 1
    malignant
    actual 0
    benign
    TNFP
    actual 1
    malignant
    FNTP

    Shaded
    tumours at or past the threshold: flagged as malignant
    Lower it
    fewer cancers missed (higher recall), more benign tumours flagged (lower precision)
    The dot
    where this threshold sits on the ROC curve

    The ROC AUC does not move with the threshold: it summarizes every threshold at once, as part 3 of the evaluation series explains. Choosing the threshold is a decision about costs, not something the model can tell you.

    What the weights say

    30 features, standardized

    odds of malignant × ew
    per standard deviation of the feature

    Orange
    pushes toward malignant: a larger value raises p
    Grey
    pushes toward benign
    Hover
    or tap a bar to read its weight

    Read single weights with care. Radius, perimeter and area measure nearly the same thing, and so do several other groups, so correlated features share the credit between them, and a weight can even point the opposite way from its feature's own link to the label. Here, compactness error rises with malignancy on its own, a correlation of +0.29, yet it gets the most negative weight of all, −0.572. And mean compactness, correlated +0.60 with malignancy, gets a weight of only 0.005: the other features already carry its information. Stronger regularization, or removing duplicate features, makes the weights easier to read.

    11

    With scikit-learn

    scikit-learn's LogisticRegression does the same job in a few lines. First on exactly the same split and scaling as run 2, so the numbers compare:

    with_sklearn.py

    from sklearn.linear_model import LogisticRegression as SkLogisticRegression
    from sklearn.metrics import (accuracy_score, confusion_matrix,
                                 precision_score, recall_score, roc_auc_score)
    
    # defaults: an L2 penalty with C=1.0, and the lbfgs solver
    sk = SkLogisticRegression()
    sk.fit(X_train, y_train)                # the same scaled data as above
    p_sk = sk.predict_proba(X_test)[:, 1]   # column 1: probability of class 1
    pred_sk = sk.predict(X_test)
    
    print("test accuracy:", round(accuracy_score(y_test, pred_sk), 4))
    print("confusion matrix:", confusion_matrix(y_test, pred_sk).tolist())
    print("precision:", round(precision_score(y_test, pred_sk), 4))
    print("recall:", round(recall_score(y_test, pred_sk), 4))
    print("ROC AUC:", round(roc_auc_score(y_test, p_sk), 4))

    output

    test accuracy: 0.9737                  # the same as ours
    confusion matrix: [[66, 1], [2, 45]]   # the same as ours
    precision: 0.9783
    recall: 0.9574
    ROC AUC: 0.9943                        # ours: 0.9933

    The predictions agree. The weights differ slightly for two reasons: scikit-learn adds an L2 penalty by default, with C = 1.0, and it solves with lbfgs (limited-memory Broyden–Fletcher–Goldfarb–Shanno), a faster method than plain gradient descent that stops at a tolerance. Give the from-scratch class the same penalty and train both to convergence, and they find the same model:

    same_model.py

    # scikit-learn minimizes  (1/2) * sum(w**2) + C * (sum of the n log losses).
    # Divide by C * n: that is our loss with l2 = 1 / (C * n). The same model.
    ours = LogisticRegression(lr=1.0, n_iters=10000, l2=1 / len(y_train))
    ours.fit(X_train, y_train)
    sk_exact = SkLogisticRegression(C=1.0, tol=1e-10, max_iter=10000)
    sk_exact.fit(X_train, y_train)
    
    gap = np.abs(ours.w - sk_exact.coef_[0]).max()
    print("largest weight difference:", gap.round(8))
    print("bias difference:", abs(ours.b - sk_exact.intercept_[0]).round(8))

    output

    largest weight difference: 8.1e-07   # the same model, to six decimal places
    bias difference: 5e-08

    In practice the split and the scaling are done by scikit-learn too. A pipeline fits the scaler on the training data only, the rule from section 07, without you having to remember it:

    pipeline.py

    from sklearn.model_selection import train_test_split
    from sklearn.pipeline import make_pipeline
    from sklearn.preprocessing import StandardScaler
    
    X_all, y_all = data.data, 1 - data.target
    X_tr, X_te, y_tr, y_te = train_test_split(
        X_all, y_all, test_size=0.2, stratify=y_all, random_state=42)
    
    # the scaler learns its mean and std from the training data only
    clf = make_pipeline(StandardScaler(), SkLogisticRegression())
    clf.fit(X_tr, y_tr)
    p_te = clf.predict_proba(X_te)[:, 1]
    print("test accuracy:", round(clf.score(X_te, y_te), 4))
    print("ROC AUC:", round(roc_auc_score(y_te, p_te), 4))

    output

    test accuracy: 0.9649   # a different split, so a different test set
    ROC AUC: 0.996
    12

    In one paragraph

    A linear score, passed through a sigmoid to get a probability, with weights learned by minimizing log loss through gradient descent.

    The gradient is Xᵀ(p − y) / n, because the sigmoid's slope cancels against log loss: dw is the input times the error, averaged, and db is the error, averaged. Training is O(n · d) per iteration and prediction is O(d). The weights read as changes in log-odds, and the threshold trades precision against recall.

    Next, part 4: multiclass logistic regression, the same model with one score per class and a softmax, trained on handwritten digits.