What is attention in Transformer models?
Attention is how, for each token, a model decides how much to look at every other token in context. The result mixes values so related words can receive higher weight.
Transformers often use multi-head attention — several “heads” that can learn different relationship types.
mej7.com provides illustrative attention maps and Q/K/V sections for intuition — not measurements from a production model.
The attention map on the mechanistic interpretability level is a toy as well. The field is explained in what mechanistic interpretability is, and the lab is at the attention map.