Query / Key / Value
Query, Key, Value
Every token produces three vectors with different roles in the attention mechanism. The angle between a Query and a Key vector determines how strongly they match.
In more detail
Each token's vector x is multiplied by three different learned matrices to produce three vectors:
- Query q = x·W_Q: what this position is looking for.
- Key k = x·W_K: what this position offers for matching.
- Value v = x·W_V: the information this position passes on if it is selected.
The match between position i and position j is the dot product q_i · k_j. A higher score means more attention. The dot product equals |q|·|k|·cos θ, so it depends on both the angle and the lengths of the vectors. In the slider above the lengths are kept fixed, which is why only the angle matters there. In real models the lengths matter too.
The scores are divided by √d_k (section 10), turned into weights with softmax, and used to mix the value vectors: the output at position i is Σ_j weight(i, j) · v_j.
Why three separate matrices? Because what a token is looking for does not have to be the same as what it offers or what it passes on. This gives the model flexibility.
"Query", "key" and "value" are helpful metaphors. In reality they are just learned linear projections, and what they encode must be found out by analysing the trained model.