Multi-Head Attention
Several attention "heads" at once
The model runs several attention mechanisms in parallel ("heads"), each potentially learning different patterns.
Different heads can learn different patterns — the examples shown here are simplified interpretations for visualization, not fixed roles proven for every model.
In more detail
Instead of one attention operation, a transformer layer runs several in parallel, called heads. Each head has its own matrices W_Q, W_K and W_V, which project the vectors to a smaller size (usually d_model divided by the number of heads; in GPT-2 small, 12 heads × 64 = 768). The outputs of all heads are put together and mixed by an output matrix W_O, and the result is added to the residual stream (section 13).
Why several heads? Each head can learn to look at different positions with a different pattern. Research has found heads that mostly look at the previous token, heads that help the model copy repeated patterns (induction heads), and heads that seem to track syntactic relations. But not every head has a clean, human-readable role, many look redundant, and what a head does can differ between models.
Simplification: the roles shown in the visualization are examples for intuition, not fixed roles proven for every model.