Transformer Math (Q·K·V)
Inside the Transformer: how Q, K and V actually multiply
A full, real walk-through of scaled dot-product attention: computing Q, K and V, transposing K, multiplying, scaling, applying the triangular causal mask, and running softmax — with real numbers, for a tiny 4-token example.
Numbers here are a small, made-up example chosen to be easy to follow by hand — real models use hundreds of dimensions and different values, but the operations (multiply, transpose, scale, mask, softmax) are exactly these.
This triangular mask is used in autoregressive (decoder-style) models like GPT, which generate text left-to-right. Encoder-style models (like BERT) typically skip this mask and let every token see the whole sequence.
In more detail
The whole operation is one formula:
Attention(Q, K, V) = softmax( Q Kᵀ / √d_k + M ) V
where M is the mask: 0 for allowed positions and −∞ for forbidden ones. The steps in the walk-through are:
- 1. Input X: one row per token.
- 2. Projections: Q = X·W_Q, K = X·W_K, V = X·W_V.
- 3. Transpose K, so that the multiplication works.
- 4. Scores S = Q Kᵀ: an n × n table; entry (i, j) is how well query i matches key j.
- 5. Scaling by √d_k. Dot products of long vectors tend to grow with their dimension d_k (if the entries are independent with variance 1, the variance of the dot product is d_k). Large scores push softmax toward an almost one-hot result with tiny gradients. Dividing by √d_k keeps the scores in a good range.
- 6. Causal mask: −∞ is added to the forbidden (future) positions, so they receive weight exactly 0 after softmax.
- 7. Softmax on each row: every row becomes weights that add up to 1.
- 8. Multiply by V: each output row is a weighted average of the value rows.
Shapes: Q and K are n × d_k, V is n × d_v, the score table is n × n, and the output is n × d_v. Because of the n × n table, the cost of attention grows with the square of the sequence length, which is why long contexts are expensive.
Simplification: the numbers in this example are small and made up so that you can follow them by hand. GPT-2 small, for example, uses d_k = 64 per head.