Machine learning from scratch · part 1Activation functions
A neuron adds its inputs up into one number. The activation function decides what comes out, mapping the whole number line onto its own range.
12 October 2026
8 min read
11 live plots
A neuron takes inputs x1 to xd, multiplies each by a weight, and adds a bias. The result is one number:
z = w1x1 + w2x2 + … + wdxd + b
z can be any real number, from −∞ to +∞. The activation function f turns it into the neuron's output, f(z), and each function squeezes or reshapes the number line its own way. The headline of each plot below says what it maps to what: the sigmoid, for one, takes every real number and returns one between 0 and 1.
Every plot runs z from −5 to 5. Drag across any of them, or use the slider, to move z on all of them at once and compare their outputs at the same input.
f(z), the output
f′(z), the slope
the slope at z
the output range
a value it approaches but never reaches
01
Why a neuron needs one
Without an activation function, a deep network is a single linear layer.
W2(W1x + b1) + b2 = (W2W1)x + (W2b1 + b2)
Two linear layers multiply out into one: a single weight matrix W2W1 and a single bias. However many are stacked, the result can only draw straight lines, or flat planes, through the data. A nonlinear f between the layers stops them collapsing and lets the network bend its boundaries.
The choice of f matters twice. Going forward, it sets which values can come out. Going backward in training, its slope f′(z) scales every gradient that flows back through the neuron, which is why every plot also shows the slope as a dashed line.
02
Output layers
The last layer's activation is set by what the network has to predict: any number, one probability, or a probability for each of several classes.
(0, 1). It gets ever closer to 0 and 1 but never reaches them
Slope
at most 0.25, at z = 0
Zero-centred
no: every output is positive, centred on 0.5
Saturates
at both ends: beyond |z| = 5 the slope is under 0.007
Used for
binary and multi-label outputs, logistic regression, and the gates inside LSTMs and GRUs
Watch out
in hidden layers its small slope makes gradients vanish (section 06). For training, PyTorch's BCEWithLogitsLoss takes z directly, which is more stable than a sigmoid followed by a log
Softmax
ℝk → (0, 1)k, summing to 1
pk = ezk / Σj ezj ∂pk/∂zj = pk(δkj − pj)
Takes
a list of k numbers, the logits, not one. Here: z from the slider, 0 and 1
Range
each output in (0, 1), and the k outputs always sum to 1
Saturates
when one logit is far above the rest, its probability approaches 1
Used for
multiclass outputs, and the attention weights inside transformers
Watch out
it is not elementwise: raising one logit lowers every other probability. Adding the same constant to all logits changes nothing, which is why code subtracts the largest first. With two classes it is the sigmoid of the difference: p1 = σ(z1 − z2)
03
The classic hidden layer
Before 2010 or so, hidden layers used the sigmoid or tanh. Tanh was usually the better of the two, because its outputs are centred on 0.
Tanh
(−∞, ∞) → (−1, 1)
f(z) = (ez − e−z) / (ez + e−z) f′(z) = 1 − f(z)²
Range
(−1, 1), never reached
Slope
at most 1, at z = 0
Zero-centred
yes, which is why it trained better than sigmoid in hidden layers
Saturates
at both ends
Used for
the hidden state of RNNs and LSTMs
Watch out
it is a stretched sigmoid, tanh(z) = 2σ(2z) − 1, so it shares the vanishing gradients
04
The ReLU family
ReLU replaced them. Above 0 its slope is exactly 1, so gradients pass through deep stacks without shrinking. The variants that follow all change what happens below 0.
ReLU (rectified linear unit)
(−∞, ∞) → [0, ∞)
f(z) = max(0, z) f′(z) = 0 if z < 0, 1 if z > 0
Range
[0, ∞): every negative z becomes exactly 0
Zero-centred
no
Saturates
flat at 0 for every negative z; never above 0
Used for
the default hidden activation in CNNs and MLPs: one comparison to compute, and gradients pass through unchanged above 0
Watch out
dying ReLU: a unit whose z is negative for every input has slope 0, so it stops learning for good. The slope at exactly 0 is undefined; libraries use 0
Leaky ReLU
(−∞, ∞) → (−∞, ∞)
f(z) = z if z > 0, αz otherwise f′(z) = 1 or α
Range
all real numbers
Zero-centred
no, though negative inputs now give small negative outputs
Saturates
never
Used for
replacing ReLU when units die, and in GAN discriminators. PReLU learns α during training
Watch out
α is one more setting to choose. PyTorch's default is 0.01, too shallow to see here, so the plot starts at 0.10
ELU (exponential linear unit)
(−∞, ∞) → (−1.00, ∞)
f(z) = z if z > 0, α(ez − 1) otherwise f′(z) = 1 or αez
Range
(−α, ∞), approaching −α for very negative z
Zero-centred
closer than ReLU, because negative inputs give negative outputs
Saturates
towards −α on the left; never on the right
Used for
some CNNs. With α = 1 the curve has no corner at 0
Watch out
an exponential for every negative input, so slower than ReLU
SELU (scaled ELU)
(−∞, ∞) → (−1.758, ∞)
f(z) = λz if z > 0, λα(ez − 1) otherwise α ≈ 1.6733, λ ≈ 1.0507
Range
(−λα, ∞) = (−1.758, ∞)
Zero-centred
by design: its constants keep activations at mean 0 and variance 1, layer after layer
Saturates
towards −1.758 on the left
Used for
self-normalising networks: deep, fully connected nets without batch normalisation
Watch out
the self-normalising property needs LeCun-normal initialisation and plain fully connected layers. Ordinary dropout breaks it; use AlphaDropout
05
Smooth activations
Modern transformers use smooth versions of ReLU. They round off the corner at 0 and let a little of the negative side through, dipping slightly below zero.
GELU (Gaussian error linear unit)
(−∞, ∞) → [−0.170, ∞)
f(z) = z · Φ(z), Φ the normal CDF f′(z) = Φ(z) + z · φ(z)
Range
[−0.170, ∞); its lowest point is at z ≈ −0.752
Zero-centred
no
Saturates
towards 0 for very negative z
Used for
BERT, GPT-2 and GPT-3, and vision transformers
Watch out
it is not monotonic: it dips below 0 before rising. Φ needs erf, so many implementations use 0.5z(1 + tanh(√(2/π)(z + 0.044715z³))), within 5 × 10⁻⁴ of the exact curve
Swish and SiLU
(−∞, ∞) → [−0.278, ∞)
f(z) = z · σ(βz); SiLU is β = 1 f′(z) = σ(βz) + βz · σ(βz)(1 − σ(βz))
Range
[−0.278/β, ∞); for SiLU the lowest point is −0.278, at z ≈ −1.278
Zero-centred
no
Saturates
towards 0 for very negative z
Used for
EfficientNet, the YOLOv5 family, and inside SwiGLU in LLaMA and PaLM
Watch out
also not monotonic, and it costs an exponential. As β grows it turns into ReLU; at β = 0 it would be the straight line z/2
outputs that must be positive, such as a predicted standard deviation or a rate
Watch out
a smooth ReLU, but slower to compute and never exactly 0, so it loses ReLU's sparsity
Not plotted, because they are not single curves: SwiGLU, the feed-forward block in LLaMA and PaLM, passes one linear projection of the input through Swish and uses it to gate another, Swish(xW) ⊙ (xV). Also seen: Mish, z · tanh(softplus(z)), in YOLOv4, and hard sigmoid and hard swish, cheap piecewise-linear versions used in MobileNetV3 on phones.
06
Vanishing gradients
Training sends each gradient backwards through every layer, and each activation multiplies it by its slope.
A slope below 1 shrinks the gradient at every layer. The sigmoid's slope is never more than 0.25, so ten sigmoid layers scale a gradient by at most 0.25¹⁰, about one millionth, and the early layers barely learn. The table gives each slope at the current z, and what ten layers of it would do. The weights multiply in too, and are left out here.
function
slope at z =
ten layers: slope¹⁰
linear
sigmoid
tanh
ReLU
leaky ReLU
ELU
SELU
GELU
Swish
softplus
Move z below 0. ReLU's slope is exactly 0 there and nothing gets through, which is how a ReLU unit dies. Leaky ReLU, ELU and the smooth functions keep some slope. Move z above 2 and the sigmoid and tanh slopes collapse while ReLU's stays at 1. SELU's slope above 0 is 1.05, so ten layers of it grow a gradient instead, one reason it is used only with its matching initialisation.
07
Which to use
where
use
why
Hidden layers, most networks
ReLU
cheap, and a slope of 1 above 0
Hidden layers, transformers
GELU, SiLU or SwiGLU
smooth, with a little negative output
Units dying under ReLU
Leaky ReLU or ELU
keep a slope below 0
Recurrent network state
tanh
bounded and centred on 0
Output: any number (regression)
linear
no limit on the range
Output: one probability, or several independent ones
sigmoid
each output in (0, 1)
Output: one of several classes
softmax
probabilities that sum to 1
Output: must be positive
softplus
smooth and always above 0
08
In code
Each function is one line of NumPy, and so is each slope that backpropagation needs.
by_hand.py
import numpy as np
from math import erf
def linear(z): return z
def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z) # = 2 * sigmoid(2 * z) - 1
def relu(z): return np.maximum(0, z)
def leaky_relu(z, a=0.01): return np.where(z > 0, z, a * z)
def elu(z, a=1.0): return np.where(z > 0, z, a * (np.exp(z) - 1))
def selu(z, a=1.6732632423543772, lam=1.0507009873554805):
return lam * np.where(z > 0, z, a * (np.exp(z) - 1))
def gelu(z): return z * 0.5 * (1 + np.vectorize(erf)(z / np.sqrt(2))) # NumPy has no erf
def silu(z): return z * sigmoid(z) # Swish with beta = 1
def softplus(z): return np.logaddexp(0, z) # log(1 + e^z), without overflow
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True)) # subtract the largest so exp cannot overflow
return e / e.sum(axis=-1, keepdims=True)
# slopes, for backpropagation
def d_sigmoid(z): s = sigmoid(z); return s * (1 - s) # at most 0.25
def d_tanh(z): return 1 - np.tanh(z) ** 2 # at most 1
def d_relu(z): return (z > 0).astype(float) # 0 at z = 0, by convention
d_softplus = sigmoid # the slope of softplus is the sigmoid
with_pytorch.py
import torch
import torch.nn.functional as F
z = torch.tensor(-1.5) # where the page starts
torch.sigmoid(z) # 0.182
torch.tanh(z) # -0.905
F.relu(z) # 0.000
F.leaky_relu(z, negative_slope=0.01) # -0.015
F.elu(z, alpha=1.0) # -0.777
F.selu(z) # -1.366
F.gelu(z) # -0.100 exact; approximate="tanh" for the GPT-2 form
F.silu(z) # -0.274
F.softplus(z) # 0.201
F.softmax(torch.tensor([-1.5, 0., 1.]), dim=0) # [0.057, 0.254, 0.690], the softmax plot at z = -1.5
# as layers inside a model
model = torch.nn.Sequential(torch.nn.Linear(4, 16), torch.nn.ReLU(), torch.nn.Linear(16, 1))
Next, part 2:loss functions, the number training makes small, and what each choice makes the model predict.