Matrix Multiplication
Neural networks multiply numbers
Matrix multiplication lets a network transform large amounts of information in a single mathematical step. Click a result cell to see exactly which row and column combined to produce it.
Amber = the row from A being used. Cyan = the column from B being used. Each result cell is the dot product of that row and that column.
In more detail
A layer of a neural network has many neurons, and each one needs a sum of products. Matrix multiplication computes all of them at once.
If A has m rows and n columns, and B has n rows and p columns, then C = AB has m rows and p columns. Each entry C[i][j] is the dot product of row i of A with column j of B: multiply the matching entries and add the results. The "inner" sizes (n) must match.
In a neural layer, the input vector x is multiplied by a weight matrix W, a bias vector is added, and the result goes through an activation function: y = f(xW + b).
Two reasons this matters:
- Speed. Graphics processors (GPUs) are built to perform huge numbers of these multiplications in parallel. Most of the computation in a transformer is matrix multiplication.
- Meaning of the dot product. A dot product is larger when two vectors point in a similar direction. This idea is at the heart of attention (sections 09 and 10).
Note that the order matters: in general AB is not the same as BA.
Simplification: the matrices here are tiny. In real models they have thousands of rows and columns.