Activation Functions

Machine learning from scratch · part 1Activation functions

A neuron adds its inputs up into one number. The activation function decides what comes out, mapping the whole number line onto its own range.

A neuron takes inputs x1 to xd, multiplies each by a weight, and adds a bias. The result is one number:

z = w1x1 + w2x2 + … + wdxd + b

z can be any real number, from −∞ to +∞. The activation function f turns it into the neuron's output, f(z), and each function squeezes or reshapes the number line its own way. The headline of each plot below says what it maps to what: the sigmoid, for one, takes every real number and returns one between 0 and 1.

Every plot runs z from −5 to 5. Drag across any of them, or use the slider, to move z on all of them at once and compare their outputs at the same input.

01

Why a neuron needs one

Without an activation function, a deep network is a single linear layer.

W2(W1x + b1) + b2 = (W2W1)x + (W2b1 + b2)

Two linear layers multiply out into one: a single weight matrix W2W1 and a single bias. However many are stacked, the result can only draw straight lines, or flat planes, through the data. A nonlinear f between the layers stops them collapsing and lets the network bend its boundaries.

The choice of f matters twice. Going forward, it sets which values can come out. Going backward in training, its slope f′(z) scales every gradient that flows back through the neuron, which is why every plot also shows the slope as a dashed line.

02

Output layers

The last layer's activation is set by what the network has to predict: any number, one probability, or a probability for each of several classes.

Linear (identity)

(−∞, ∞) → (−∞, ∞)

f(z) = z
f′(z) = 1

Range
all real numbers
Zero-centred
yes
Saturates
never
Used for
the output of a regression model, such as part 13 of the evaluation series, where the model estimates an ejection fraction
Watch out
in a hidden layer it adds nothing: see section 01

Sigmoid (logistic)

(−∞, ∞) → (0, 1)

f(z) = 1 / (1 + e−z)
f′(z) = f(z) · (1 − f(z))

Range
(0, 1). It gets ever closer to 0 and 1 but never reaches them
Slope
at most 0.25, at z = 0
Zero-centred
no: every output is positive, centred on 0.5
Saturates
at both ends: beyond |z| = 5 the slope is under 0.007
Used for
binary and multi-label outputs, logistic regression, and the gates inside LSTMs and GRUs
Watch out
in hidden layers its small slope makes gradients vanish (section 06). For training, PyTorch's BCEWithLogitsLoss takes z directly, which is more stable than a sigmoid followed by a log

Softmax

ℝk → (0, 1)k, summing to 1

pk = ezk / Σj ezj
∂pk/∂zj = pk(δkj − pj)

Takes
a list of k numbers, the logits, not one. Here: z from the slider, 0 and 1
Range
each output in (0, 1), and the k outputs always sum to 1
Saturates
when one logit is far above the rest, its probability approaches 1
Used for
multiclass outputs, and the attention weights inside transformers
Watch out
it is not elementwise: raising one logit lowers every other probability. Adding the same constant to all logits changes nothing, which is why code subtracts the largest first. With two classes it is the sigmoid of the difference: p1 = σ(z1 − z2)
03

The classic hidden layer

Before 2010 or so, hidden layers used the sigmoid or tanh. Tanh was usually the better of the two, because its outputs are centred on 0.

Tanh

(−∞, ∞) → (−1, 1)

f(z) = (ez − e−z) / (ez + e−z)
f′(z) = 1 − f(z)²

Range
(−1, 1), never reached
Slope
at most 1, at z = 0
Zero-centred
yes, which is why it trained better than sigmoid in hidden layers
Saturates
at both ends
Used for
the hidden state of RNNs and LSTMs
Watch out
it is a stretched sigmoid, tanh(z) = 2σ(2z) − 1, so it shares the vanishing gradients
04

The ReLU family

ReLU replaced them. Above 0 its slope is exactly 1, so gradients pass through deep stacks without shrinking. The variants that follow all change what happens below 0.

ReLU (rectified linear unit)

(−∞, ∞) → [0, ∞)

f(z) = max(0, z)
f′(z) = 0 if z < 0, 1 if z > 0

Range
[0, ∞): every negative z becomes exactly 0
Zero-centred
no
Saturates
flat at 0 for every negative z; never above 0
Used for
the default hidden activation in CNNs and MLPs: one comparison to compute, and gradients pass through unchanged above 0
Watch out
dying ReLU: a unit whose z is negative for every input has slope 0, so it stops learning for good. The slope at exactly 0 is undefined; libraries use 0

Leaky ReLU

(−∞, ∞) → (−∞, ∞)

f(z) = z if z > 0, αz otherwise
f′(z) = 1 or α

Range
all real numbers
Zero-centred
no, though negative inputs now give small negative outputs
Saturates
never
Used for
replacing ReLU when units die, and in GAN discriminators. PReLU learns α during training
Watch out
α is one more setting to choose. PyTorch's default is 0.01, too shallow to see here, so the plot starts at 0.10

ELU (exponential linear unit)

(−∞, ∞) → (−1.00, ∞)

f(z) = z if z > 0, α(ez − 1) otherwise
f′(z) = 1 or αez

Range
(−α, ∞), approaching −α for very negative z
Zero-centred
closer than ReLU, because negative inputs give negative outputs
Saturates
towards −α on the left; never on the right
Used for
some CNNs. With α = 1 the curve has no corner at 0
Watch out
an exponential for every negative input, so slower than ReLU

SELU (scaled ELU)

(−∞, ∞) → (−1.758, ∞)

f(z) = λz if z > 0, λα(ez − 1) otherwise
α ≈ 1.6733, λ ≈ 1.0507

Range
(−λα, ∞) = (−1.758, ∞)
Zero-centred
by design: its constants keep activations at mean 0 and variance 1, layer after layer
Saturates
towards −1.758 on the left
Used for
self-normalising networks: deep, fully connected nets without batch normalisation
Watch out
the self-normalising property needs LeCun-normal initialisation and plain fully connected layers. Ordinary dropout breaks it; use AlphaDropout
05

Smooth activations

Modern transformers use smooth versions of ReLU. They round off the corner at 0 and let a little of the negative side through, dipping slightly below zero.

GELU (Gaussian error linear unit)

(−∞, ∞) → [−0.170, ∞)

f(z) = z · Φ(z), Φ the normal CDF
f′(z) = Φ(z) + z · φ(z)

Range
[−0.170, ∞); its lowest point is at z ≈ −0.752
Zero-centred
no
Saturates
towards 0 for very negative z
Used for
BERT, GPT-2 and GPT-3, and vision transformers
Watch out
it is not monotonic: it dips below 0 before rising. Φ needs erf, so many implementations use 0.5z(1 + tanh(√(2/π)(z + 0.044715z³))), within 5 × 10⁻⁴ of the exact curve

Swish and SiLU

(−∞, ∞) → [−0.278, ∞)

f(z) = z · σ(βz); SiLU is β = 1
f′(z) = σ(βz) + βz · σ(βz)(1 − σ(βz))

Range
[−0.278/β, ∞); for SiLU the lowest point is −0.278, at z ≈ −1.278
Zero-centred
no
Saturates
towards 0 for very negative z
Used for
EfficientNet, the YOLOv5 family, and inside SwiGLU in LLaMA and PaLM
Watch out
also not monotonic, and it costs an exponential. As β grows it turns into ReLU; at β = 0 it would be the straight line z/2

Softplus

(−∞, ∞) → (0, ∞)

f(z) = log(1 + ez)
f′(z) = σ(z), exactly the sigmoid

Range
(0, ∞): never exactly 0
Zero-centred
no
Saturates
towards 0 for very negative z
Used for
outputs that must be positive, such as a predicted standard deviation or a rate
Watch out
a smooth ReLU, but slower to compute and never exactly 0, so it loses ReLU's sparsity

Not plotted, because they are not single curves: SwiGLU, the feed-forward block in LLaMA and PaLM, passes one linear projection of the input through Swish and uses it to gate another, Swish(xW) ⊙ (xV). Also seen: Mish, z · tanh(softplus(z)), in YOLOv4, and hard sigmoid and hard swish, cheap piecewise-linear versions used in MobileNetV3 on phones.

06

Vanishing gradients

Training sends each gradient backwards through every layer, and each activation multiplies it by its slope.

A slope below 1 shrinks the gradient at every layer. The sigmoid's slope is never more than 0.25, so ten sigmoid layers scale a gradient by at most 0.25¹⁰, about one millionth, and the early layers barely learn. The table gives each slope at the current z, and what ten layers of it would do. The weights multiply in too, and are left out here.

functionslope at z = ten layers: slope¹⁰
linear
sigmoid
tanh
ReLU
leaky ReLU
ELU
SELU
GELU
Swish
softplus

Move z below 0. ReLU's slope is exactly 0 there and nothing gets through, which is how a ReLU unit dies. Leaky ReLU, ELU and the smooth functions keep some slope. Move z above 2 and the sigmoid and tanh slopes collapse while ReLU's stays at 1. SELU's slope above 0 is 1.05, so ten layers of it grow a gradient instead, one reason it is used only with its matching initialisation.

07

Which to use

whereusewhy
Hidden layers, most networksReLUcheap, and a slope of 1 above 0
Hidden layers, transformersGELU, SiLU or SwiGLUsmooth, with a little negative output
Units dying under ReLULeaky ReLU or ELUkeep a slope below 0
Recurrent network statetanhbounded and centred on 0
Output: any number (regression)linearno limit on the range
Output: one probability, or several independent onessigmoideach output in (0, 1)
Output: one of several classessoftmaxprobabilities that sum to 1
Output: must be positivesoftplussmooth and always above 0
08

In code

Each function is one line of NumPy, and so is each slope that backpropagation needs.

by_hand.py

import numpy as np
from math import erf

def linear(z):     return z
def sigmoid(z):    return 1 / (1 + np.exp(-z))
def tanh(z):       return np.tanh(z)                       # = 2 * sigmoid(2 * z) - 1
def relu(z):       return np.maximum(0, z)
def leaky_relu(z, a=0.01): return np.where(z > 0, z, a * z)
def elu(z, a=1.0):         return np.where(z > 0, z, a * (np.exp(z) - 1))
def selu(z, a=1.6732632423543772, lam=1.0507009873554805):
    return lam * np.where(z > 0, z, a * (np.exp(z) - 1))
def gelu(z):       return z * 0.5 * (1 + np.vectorize(erf)(z / np.sqrt(2)))   # NumPy has no erf
def silu(z):       return z * sigmoid(z)                   # Swish with beta = 1
def softplus(z):   return np.logaddexp(0, z)               # log(1 + e^z), without overflow
def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))          # subtract the largest so exp cannot overflow
    return e / e.sum(axis=-1, keepdims=True)

# slopes, for backpropagation
def d_sigmoid(z):  s = sigmoid(z); return s * (1 - s)      # at most 0.25
def d_tanh(z):     return 1 - np.tanh(z) ** 2              # at most 1
def d_relu(z):     return (z > 0).astype(float)            # 0 at z = 0, by convention
d_softplus = sigmoid                                        # the slope of softplus is the sigmoid

with_pytorch.py

import torch
import torch.nn.functional as F

z = torch.tensor(-1.5)                  # where the page starts
torch.sigmoid(z)                        # 0.182
torch.tanh(z)                           # -0.905
F.relu(z)                               # 0.000
F.leaky_relu(z, negative_slope=0.01)    # -0.015
F.elu(z, alpha=1.0)                     # -0.777
F.selu(z)                               # -1.366
F.gelu(z)                               # -0.100  exact; approximate="tanh" for the GPT-2 form
F.silu(z)                               # -0.274
F.softplus(z)                           # 0.201
F.softmax(torch.tensor([-1.5, 0., 1.]), dim=0)   # [0.057, 0.254, 0.690], the softmax plot at z = -1.5

# as layers inside a model
model = torch.nn.Sequential(torch.nn.Linear(4, 16), torch.nn.ReLU(), torch.nn.Linear(16, 1))

Next, part 2: loss functions, the number training makes small, and what each choice makes the model predict.