Machine learning from scratch · part 3Logistic regression
A model for yes-or-no questions. It predicts the probability that an example belongs to class 1, from a weighted sum of its features passed through a sigmoid. Despite the name, it classifies.
12 October 2026
20 min read
6 live plots
Logistic regression is built from two pieces covered earlier in the series: the sigmoid activation from part 1 and the binary cross-entropy loss from part 2. Both are written out again below where they are needed. The page builds the model in four steps, derives its gradient, writes it in NumPy from scratch, runs it on two datasets and checks it against scikit-learn.
01
What it does
Logistic regression predicts the probability that an example belongs to class 1.
p = σ(w · x + b)
Class 1 is whichever outcome you want to detect: fraud or not, default or not, malignant or benign. Despite the name, the model is used for classification, not regression; the name comes from the logistic function, another name for the sigmoid. It is widely used in banking and medicine because it is fast, simple, and its weights are easy to explain.
Input. A feature matrix X of shape (n, d): n examples with d features each, such as income and age. For training, also the labels y: n values, each 0 or 1.
Output. For each example, a probability p between 0 and 1 that it is class 1. A threshold, usually 0.5, turns it into a class.
What it learns. A weight vector w, one weight per feature, and a bias b.
02
Step 1: a linear score
Combine the features into one number with a weighted sum, as in linear regression:
z = w₁x₁ + w₂x₂ + … + wdxd + b in NumPy: z = X @ w + b
z can be any number, from −∞ to ∞, so it is not yet a probability. But it already decides the class: the model will say class 1 when z > 0 and class 0 when z < 0. The points where z = 0 form the decision boundary. With two features that is a straight line, and in more dimensions a flat plane, so logistic regression separates classes linearly.
The plot below holds the 320 training points of the first dataset in section 09: two overlapping clusters with two features each. Move the weights and watch the boundary turn and slide.
The boundary, by hand
z = w₁x₁ + w₂x₂ + b
class 1
class 0
boundary, p = 0.5
p = 0.1 and 0.9
Shading
orange where p is above 0.5, grey below; the deeper the colour, the more certain
w
the arrow: it points across the boundary toward class 1. Doubling it keeps the line but sharpens the change from 0 to 1
b
slides the line without turning it
Press the trained weights: the boundary runs diagonally between the two clusters, and a few points of each class still sit on the wrong side. The clusters overlap, so no straight line can separate them perfectly.
03
Step 2: squash it into a probability
The sigmoid function, from part 1, maps any z into the range 0 to 1:
σ(z) = 1 / (1 + e−z)
z = −5 → 0.007 almost surely class 0 z = 0 → 0.5 unsure z = 5 → 0.993 almost surely class 1
The sigmoid is exactly 0.5 at z = 0, so the boundary where p = 0.5 is the same line where z = 0. Run the sigmoid backwards and z turns out to be the log-odds: z = log(p / (1 − p)). The model is linear in the log-odds, and that is what makes its weights readable: each weight is the change in log-odds per unit of its feature.
Every point, through the sigmoid
(−∞, ∞) → (0, 1)
class 1
class 0
p = σ(z) = 1 / (1 + e−z) z = log(p / (1 − p))
The dots
the 320 training points, each at its own z for the weights set in the card above. Change them there and the dots move here
Good weights
put the orange dots high on the right and the hollow ones low on the left
04
Step 3: measure the error with log loss
Training needs one number that says how wrong the predictions are. Logistic regression uses log loss, also called binary cross-entropy, from part 2:
loss = −mean( y · log(p) + (1 − y) · log(1 − p) )
If the true label is 1, the second term is multiplied by 0 and only −log(p) counts: small when p is near 1, huge when p is near 0. If the label is 0, only −log(1 − p) counts. So confident wrong answers are punished heavily.
Why not mean squared error (MSE)? It works for linear regression, but put a sigmoid in front of it and two things go wrong. The loss is no longer convex, so it can have flat stretches and local minima where gradient descent stalls. And its gradient vanishes on confidently wrong examples. Log loss is convex in w and b, a single bowl, so gradient descent reliably finds the best weights. The card plots both losses for one example whose true class is 1.
Log loss against squared error
one example, true class 1
log loss, −log p
squared error, (1 − p)²
its gradient, p − 1
its gradient, −2p(1 − p)²
Log loss
keeps growing as the answer gets more confidently wrong, and its gradient stays near −1: the worst mistakes get the strongest correction
Squared error
can never exceed 1, and its gradient fades to 0 on the far left: the model stops learning from exactly the examples it gets most wrong
05
Step 4: learn the weights with gradient descent
Start with every weight at 0. Then repeat: compute the predictions, and move each weight a small step in the direction that lowers the loss. The gradient of log loss simplifies nicely; section 06 derives it.
error = p - y # prediction minus truth
dw = X.T @ error / n # gradient for the weights
db = np.mean(error) # gradient for the bias
w = w - lr * dw # step downhill
b = b - lr * db
To remember dw = the input times the error, averaged: Xᵀ(p − y) / n db = the error, averaged: mean(p − y)
The bias works like a weight on an input that is always 1, so its gradient is the error alone.
The learning rate lr sets the step size, and the number of iterations sets how long training runs. The card runs exactly this loop on the 320 points.
Gradient descent, step by step
w ← w − lr · dw
class 1
class 0
boundary, p = 0.5
p = 0.1 and 0.9
Start
every weight at 0, so every p is 0.5 and the loss is log 2 = 0.693
Each step
the gradient is computed on all 320 points, then w and b move against it
lr
changing it starts a new run, and the last run stays on the loss plot as a dashed line
At lr = 0.1, 1000 steps land on w = (2.923, 2.533) and b ≈ 0, with a loss of 0.1777: the same numbers the code prints in section 09. Now try other rates. At 0.01 the loss is still falling after 1000 steps: too slow. At 10 it is done within five steps. At 100 each step overshoots, and the loss jumps up and down before it settles: too big.
06
Where the gradient comes from
Gradient descent needs the gradient of the loss with respect to every weight. Here it is derived for one example, with its linear score z, its predicted probability p and its cross-entropy loss, using the natural log:
z = w · x + b p = σ(z) = 1 / (1 + e−z) loss = −[ y log p + (1 − y) log(1 − p) ]
The loss depends on w only through p, and p only through z, so the chain rule splits the gradient into three factors:
1. The loss with respect to p. Use d/dp log(p) = 1/p and d/dp log(1 − p) = −1/(1 − p), then put both fractions over the common denominator p(1 − p). The yp terms cancel:
2. The sigmoid with respect to z. Write p = (1 + e−z)−1 and differentiate. The last step uses e−z / (1 + e−z) = 1 − p. This is the sigmoid's slope, plotted on its card in part 1:
4. The score with respect to each weight. Since z = w₁x₁ + w₂x₂ + … + b, each weight's derivative is its own feature:
∂z/∂wj = xj ∂z/∂b = 1
5. The gradients for one example:
∂loss/∂wj = (p − y) · xj ∂loss/∂b = p − y
6. Average over all n examples. The total loss is an average over the n examples, so the gradients are averages too. Vectorized, this is exactly the code:
dw = (1/n) Σi (pi − yi) xi = Xᵀ (p − y) / n db = (1/n) Σi (pi − yi) = mean(p − y)
The update is then w = w − lr · dw and b = b − lr · db.
With L2 regularization, a penalty on the squared weights, the loss gets an extra term (l2 / 2) · Σ wj². Its derivative adds l2 · w to dw: that is the + self.l2 * self.w in the code.
So error = p − y is not the loss. It is each example's contribution to the gradient. The loss itself, with the logs, is computed separately only to track progress: the weights would train exactly the same without it.
07
Practical details
Scale the features. Subtract the mean and divide by the standard deviation, using the training set's statistics only, so the test set stays unseen. Without it, a feature measured in thousands dominates one measured in fractions, and one learning rate cannot suit both, so training converges slowly.
Regularize. L2 regularization adds l2 · w to the gradient, which keeps the weights small and reduces overfitting. scikit-learn sets the same penalty with C, which works the other way round: l2 = 1 / (C · n), so a smaller C means a stronger penalty. Section 11 shows the two giving the same model.
Clip z. e−z overflows a 64-bit float once −z passes about 709, so the code clips z to ±500 first. The sigmoid of ±500 is 0 or 1 to every digit a float can hold, so nothing is lost.
Read the weights. A positive weight pushes toward class 1 and a negative one toward class 0. Each weight is the change in log-odds per unit of its feature, so ew is the factor the odds are multiplied by. With standardized features, the unit is one standard deviation.
Choose the threshold. 0.5 is only a default. In fraud detection, a lower threshold catches more fraud, a higher recall, at the cost of more false alarms, a lower precision. Section 10 lets you move it on real data.
Time and space. One training step computes X @ w and Xᵀ · error, each O(n · d), so k steps cost O(k · n · d). Predicting one example costs O(d), and the model stores only its d weights and a bias: O(d) space.
08
The model, from scratch
The whole model is one class in NumPy. fit is the loop from section 05, and the comments point back to the steps above.
logistic_regression.py
import numpy as np
class LogisticRegression:
"""p = sigmoid(X @ w + b), trained by gradient descent on log loss."""
def __init__(self, lr=0.1, n_iters=1000, l2=0.0):
self.lr = lr # learning rate: the size of each step
self.n_iters = n_iters # how many steps to take
self.l2 = l2 # L2 regularization strength; 0 = none
self.w = None # one weight per feature, set by fit
self.b = 0.0 # the bias
self.losses = [] # the loss at every step, to track training
def _sigmoid(self, z):
z = np.clip(z, -500, 500) # keep exp(-z) from overflowing
return 1 / (1 + np.exp(-z)) # any number -> between 0 and 1
def _loss(self, y, p):
eps = 1e-15
p = np.clip(p, eps, 1 - eps) # log(0) would be minus infinity
# binary cross-entropy, plus the L2 penalty
bce = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
return bce + (self.l2 / 2) * np.sum(self.w ** 2)
def fit(self, X, y):
n_samples, n_features = X.shape
self.w = np.zeros(n_features) # start at 0: every p starts at 0.5
self.b = 0.0
self.losses = []
for _ in range(self.n_iters):
# 1. predict: the linear score, then the sigmoid
p = self._sigmoid(X @ self.w + self.b)
# 2. record the loss, only to track progress
self.losses.append(self._loss(y, p))
# 3. the gradients; error is each example's share of them
error = p - y
dw = X.T @ error / n_samples + self.l2 * self.w
db = np.mean(error)
# 4. step downhill
self.w -= self.lr * dw
self.b -= self.lr * db
return self
def predict_proba(self, X):
# the probability of class 1
return self._sigmoid(X @ self.w + self.b)
def predict(self, X, threshold=0.5):
# probability -> class 0 or 1
return (self.predict_proba(X) >= threshold).astype(int)
09
Run 1: two clusters
Two overlapping clusters of 200 points each, made with NumPy alone: the 320 training points plotted in sections 02 to 05. The model ends with a loss of 0.1777 and a test accuracy of 0.8875, 71 of 80.
showhide the code and its output
The code makes the two clusters, splits them 80/20, scales them, trains and evaluates.
run_clusters.py
# 1. Make data: 2 features, 2 classes
np.random.seed(42)
X0 = np.random.randn(200, 2) - 1 # class 0, centred at (-1, -1)
X1 = np.random.randn(200, 2) + 1 # class 1, centred at (1, 1)
X = np.vstack([X0, X1])
y = np.array([0] * 200 + [1] * 200)
# 2. Shuffle and split 80% train, 20% test
idx = np.random.permutation(len(X))
X, y = X[idx], y[idx]
split = int(0.8 * len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]
# 3. Standardize with training statistics only
mean, std = X_train.mean(axis=0), X_train.std(axis=0)
X_train = (X_train - mean) / std
X_test = (X_test - mean) / std
# 4. Train
model = LogisticRegression(lr=0.1, n_iters=1000)
model.fit(X_train, y_train)
# 5. Evaluate
print("first loss:", round(model.losses[0], 4))
print("final loss:", round(model.losses[-1], 4))
print("weights:", model.w.round(3), "bias:", round(model.b, 3))
print("test accuracy:", np.mean(model.predict(X_test) == y_test))
print("probabilities:", model.predict_proba(X_test[:3]).round(3))
output
first loss: 0.6931 # log 2: every p starts at 0.5
final loss: 0.1777 # the model learned
weights: [2.923 2.533] bias: -0.0 # both push toward class 1
test accuracy: 0.8875 # 71 of 80: the two clusters overlap
probabilities: [0.324 0.435 0.995] # for the first three test points
A falling loss confirms that training works: it starts at log 2 = 0.693, where every prediction is 0.5, and ends at 0.1777. Both weights are positive because class 1 sits up and to the right. The bias prints as −0.0, a tiny negative number rounded: the classes are balanced, so the boundary passes almost through the centre. The accuracy is 71 of 80 rather than all of them because the clusters overlap: no straight line can separate them perfectly.
10
Run 2: breast cancer
A standard real dataset: 569 breast tumours, 212 malignant and 357 benign, with 30 measurements each. The same class, unchanged, reaches a test accuracy of 0.9737 and misses 2 of the 47 cancers in the test set. Inside are the code and its output, a threshold you can move, and the weights the model learned.
showhide the code, output, threshold and weights
The dataset ships with scikit-learn. Each tumour is described by 30 measurements of the cell nuclei in a digitized image of a needle biopsy: the mean, the standard error and the worst (largest) value of ten properties such as radius, texture and concavity. scikit-learn labels malignant as 0; the code flips it, so that class 1 is the cancer the model should catch.
from sklearn.datasets import load_breast_cancer # only to load the data
# 1. Load: 569 tumours, 30 measurements each
data = load_breast_cancer()
X = data.data
y = 1 - data.target # malignant is 0 in scikit-learn; flip so 1 = malignant
# 2. Shuffle and split 80% train, 20% test
np.random.seed(42)
idx = np.random.permutation(len(X))
X, y = X[idx], y[idx]
split = int(0.8 * len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]
# 3. Standardize with training statistics only
mean, std = X_train.mean(axis=0), X_train.std(axis=0)
X_train = (X_train - mean) / std
X_test = (X_test - mean) / std
# 4. Train, with the same settings as before
model = LogisticRegression(lr=0.1, n_iters=1000)
model.fit(X_train, y_train)
# 5. Evaluate on the test set, by hand
p = model.predict_proba(X_test)
pred = model.predict(X_test)
tp = int(np.sum((pred == 1) & (y_test == 1))) # malignant, flagged
fn = int(np.sum((pred == 0) & (y_test == 1))) # malignant, missed
fp = int(np.sum((pred == 1) & (y_test == 0))) # benign, flagged
tn = int(np.sum((pred == 0) & (y_test == 0))) # benign, cleared
# ROC AUC: the chance that a malignant tumour outscores a benign one
auc = np.mean(p[y_test == 1][:, None] > p[y_test == 0][None, :])
print("first loss:", round(model.losses[0], 4))
print("final loss:", round(model.losses[-1], 4))
print("test accuracy:", round(np.mean(pred == y_test), 4))
print("confusion matrix:", [[tn, fp], [fn, tp]])
print("precision:", round(tp / (tp + fp), 4))
print("recall:", round(tp / (tp + fn), 4))
print("ROC AUC:", round(auc, 4))
top = np.argsort(model.w)[::-1][:3]
print("strongest push to malignant:", ", ".join(data.feature_names[top]))
output
first loss: 0.6931 # log 2 again
final loss: 0.0523
test accuracy: 0.9737 # 111 of 114
confusion matrix: [[66, 1], [2, 45]] # rows: truth; columns: prediction
precision: 0.9783 # 45 of the 46 flagged are malignant
recall: 0.9574 # 45 of the 47 malignant are caught
ROC AUC: 0.9933
strongest push to malignant: worst texture, radius error, worst radius
The 2 malignant tumours scored below 0.5 are the costly mistake: a missed cancer. The threshold is the lever. Lower it and those cases get flagged, at the price of more benign tumours sent for a second look. The card shows all 114 test tumours, each at the probability the model gave it.
Choosing the threshold
114 test tumours
ROC curve
random guessing
pred. 0 benign
pred. 1 malignant
actual 0 benign
TN
FP
actual 1 malignant
FN
TP
Shaded
tumours at or past the threshold: flagged as malignant
The ROC AUC does not move with the threshold: it summarizes every threshold at once, as part 3 of the evaluation series explains. Choosing the threshold is a decision about costs, not something the model can tell you.
What the weights say
30 features, standardized
odds of malignant × ew per standard deviation of the feature
Orange
pushes toward malignant: a larger value raises p
Grey
pushes toward benign
Hover
or tap a bar to read its weight
Read single weights with care. Radius, perimeter and area measure nearly the same thing, and so do several other groups, so correlated features share the credit between them, and a weight can even point the opposite way from its feature's own link to the label. Here, compactness error rises with malignancy on its own, a correlation of +0.29, yet it gets the most negative weight of all, −0.572. And mean compactness, correlated +0.60 with malignancy, gets a weight of only 0.005: the other features already carry its information. Stronger regularization, or removing duplicate features, makes the weights easier to read.
11
With scikit-learn
scikit-learn's LogisticRegression does the same job in a few lines. First on exactly the same split and scaling as run 2, so the numbers compare:
with_sklearn.py
from sklearn.linear_model import LogisticRegression as SkLogisticRegression
from sklearn.metrics import (accuracy_score, confusion_matrix,
precision_score, recall_score, roc_auc_score)
# defaults: an L2 penalty with C=1.0, and the lbfgs solver
sk = SkLogisticRegression()
sk.fit(X_train, y_train) # the same scaled data as above
p_sk = sk.predict_proba(X_test)[:, 1] # column 1: probability of class 1
pred_sk = sk.predict(X_test)
print("test accuracy:", round(accuracy_score(y_test, pred_sk), 4))
print("confusion matrix:", confusion_matrix(y_test, pred_sk).tolist())
print("precision:", round(precision_score(y_test, pred_sk), 4))
print("recall:", round(recall_score(y_test, pred_sk), 4))
print("ROC AUC:", round(roc_auc_score(y_test, p_sk), 4))
output
test accuracy: 0.9737 # the same as ours
confusion matrix: [[66, 1], [2, 45]] # the same as ours
precision: 0.9783
recall: 0.9574
ROC AUC: 0.9943 # ours: 0.9933
The predictions agree. The weights differ slightly for two reasons: scikit-learn adds an L2 penalty by default, with C = 1.0, and it solves with lbfgs (limited-memory Broyden–Fletcher–Goldfarb–Shanno), a faster method than plain gradient descent that stops at a tolerance. Give the from-scratch class the same penalty and train both to convergence, and they find the same model:
same_model.py
# scikit-learn minimizes (1/2) * sum(w**2) + C * (sum of the n log losses).
# Divide by C * n: that is our loss with l2 = 1 / (C * n). The same model.
ours = LogisticRegression(lr=1.0, n_iters=10000, l2=1 / len(y_train))
ours.fit(X_train, y_train)
sk_exact = SkLogisticRegression(C=1.0, tol=1e-10, max_iter=10000)
sk_exact.fit(X_train, y_train)
gap = np.abs(ours.w - sk_exact.coef_[0]).max()
print("largest weight difference:", gap.round(8))
print("bias difference:", abs(ours.b - sk_exact.intercept_[0]).round(8))
output
largest weight difference: 8.1e-07 # the same model, to six decimal places
bias difference: 5e-08
In practice the split and the scaling are done by scikit-learn too. A pipeline fits the scaler on the training data only, the rule from section 07, without you having to remember it:
pipeline.py
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X_all, y_all = data.data, 1 - data.target
X_tr, X_te, y_tr, y_te = train_test_split(
X_all, y_all, test_size=0.2, stratify=y_all, random_state=42)
# the scaler learns its mean and std from the training data only
clf = make_pipeline(StandardScaler(), SkLogisticRegression())
clf.fit(X_tr, y_tr)
p_te = clf.predict_proba(X_te)[:, 1]
print("test accuracy:", round(clf.score(X_te, y_te), 4))
print("ROC AUC:", round(roc_auc_score(y_te, p_te), 4))
output
test accuracy: 0.9649 # a different split, so a different test set
ROC AUC: 0.996
12
In one paragraph
A linear score, passed through a sigmoid to get a probability, with weights learned by minimizing log loss through gradient descent.
The gradient is Xᵀ(p − y) / n, because the sigmoid's slope cancels against log loss: dw is the input times the error, averaged, and db is the error, averaged. Training is O(n · d) per iteration and prediction is O(d). The weights read as changes in log-odds, and the threshold trades precision against recall.
Next, part 4:multiclass logistic regression, the same model with one score per class and a softmax, trained on handwritten digits.