Activation Functions
Activation functions
After each multiplication, the result passes through an activation function, which lets the network learn non-linear patterns.
In more detail
Without activation functions, a stack of layers would collapse into one single linear transformation, no matter how many layers it had. An activation function applies a simple non-linear function to each number, and this is what lets networks learn curved, complex relationships.
- ReLU: f(x) = max(0, x). Negative values become zero; positive values pass unchanged. Simple and fast.
- Sigmoid: f(x) = 1 / (1 + e^(−x)). Squashes any number into the range 0 to 1.
- Tanh: squashes numbers into the range −1 to 1.
Sigmoid and tanh become very flat for large positive or negative inputs, which can slow learning. ReLU avoids this for positive inputs, but a neuron can get stuck outputting zero for everything.
Large transformer models mostly use smoother variants. GPT-2 and BERT use GELU, and many recent models use gated variants such as SwiGLU. The principle is the same: a non-linear step after each linear one.