Neural Network Lab
See the network compute
Full interactive feed-forward laboratory — forward pass, matrix multiply, training on real toy data, and neuron autopsy. All numbers come from the actual simulation.
In more detail
A neural network is a stack of layers. Each layer takes a vector of numbers, multiplies it by a matrix of weights, adds biases and applies an activation function. The output becomes the input of the next layer. "Feed-forward" means that information flows in one direction, from input to output. This process is called the forward pass.
Training adjusts the weights so that the outputs match the desired targets on many examples (sections 17 and 18). A network with at least one hidden layer and a non-linear activation can approximate any continuous function on a bounded region to any desired accuracy, given enough neurons (the universal approximation theorem). That theorem says it is possible, not that it is easy to find the right weights.
Transformers contain feed-forward networks too: the MLP in every block (section 13). The lab below uses a tiny network on a toy problem. Real language models use the same building blocks at a vastly larger scale.