Transformer

A transformer is a stack of identical blocks. The original 2017 design had an encoder and a decoder; GPT-style chat models use only the decoder part ("decoder-only"). Each block has two sublayers: multi-head self-attention and a feed-forward network (MLP).

• Residual connections. The output of each sublayer is added to its input instead of replacing it. This gives every position a running vector, the residual stream, that each layer reads from and adds to. It also makes deep networks much easier to train.
• Layer normalization. Rescales each token's vector to keep the numbers in a stable range. The 2017 design applied it after each sublayer; GPT-2 and most later models apply it before ("pre-LN"), which tends to make training more stable.
• Attention sublayer. Moves information between positions.
• MLP sublayer. Two linear layers with a non-linear activation between them. It first widens the vector (typically by a factor of four: 768 to 3,072 in GPT-2 small) and then narrows it again. It is applied to each position separately. Research suggests that MLP layers act partly like stores of learned associations (Geva et al., 2021), although this is an active area of research.

After the last block, a final normalization and the unembedding matrix turn the vector of the last position into one score for every token in the vocabulary (section 14). GPT-2 small has 12 blocks and about 124 million parameters; larger models have more blocks and wider vectors.

Simplification: the diagram shows the main parts. Real models differ in details such as the normalization type, activation function and position scheme.