Positional Information

Attention on its own has no sense of order. If the input tokens were permuted and nothing else changed, the outputs would simply be permuted in the same way. But order carries meaning ("the dog bit the man" is not "the man bit the dog"), so models add positional information.

• The original 2017 Transformer added fixed sinusoidal position signals to the token embeddings.
• GPT-2 added learned position embeddings, one learned vector for each position up to its limit of 1,024.
• Many recent models use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position, so that attention scores depend on how far apart two tokens are.
• Other schemes exist. The choice affects how well a model handles texts longer than those it saw during training.

In decoder-only models, the causal mask can also let the model infer some positional information implicitly. The details differ from model to model, but the shared idea is that position must be supplied, because attention does not provide it by itself.