Residual Stream
How an input travels through the residual stream
The whole input “The cat sat on the” is embedded and climbs the residual stream as a stack of vectors, one row per token. At each layer every head reads the same stack in parallel, then all head outputs are added back into one single result. A Layer 2 head only reads a 64-dimensional window of it: move the window and see what gets through.
Every token position has its own residual stream; the drawing shows them together as one stack. Attention is the only step where positions exchange information. All dimension numbers and percentages here are illustrative guesses: in a real model the exact subspaces are not known in advance and have to be measured from the weights. Simplified: 4 heads per layer are shown (a real model has 12 or more), and each head’s subspace is drawn as a 64-dimensional segment on a line of 768 dimensions. In reality subspaces are directions in the 768-dimensional space and the overlap is measured with a dot product, but the idea is the same: if the directions are perpendicular, nothing gets through. The token embedding itself is ignored when computing the overlap.
Toy model for intuition, not real model weights.
What you are seeing
The input is the unfinished sentence "The cat sat on the". Each token is a row. The animation treats those rows as one stack of vectors and walks the stack from the embedding, through a first layer, into a second layer. At each layer the heads read the stack and their outputs are added back, which is the residual rule: add, do not replace. A control lets you pretend that layer 1 wrote into only some of 768 dimensions, and that one layer-2 head reads a 64-dimensional window. If the window misses the written band, the readout shown on the page goes quiet.
That last interaction is the point of the toy. Heads do not each receive a private copy of the whole model. They read and write directions inside one shared stream. If two subspaces are perpendicular, the dot product is zero and the later head does not see the earlier write. The page says this in its own disclaimer, and it is the part worth remembering.
What is simplified here
768 and 64 are the widths of GPT-2 small, used here as familiar round numbers, not as a measurement of this animation. The "about 567 dimensions" slider starts at a guess. The page labels it as a guess. No overlap in the drawing was computed from weights. Four heads are shown; GPT-2 small has twelve in a layer, and other models differ. The subspaces are drawn as segments of a line. In the real space they are directions, and the overlap is a dot product, which can be partial rather than a hard window. The token embedding is left out of the overlap calculation, as the on-page note says.
The motion is a lesson, not a forward pass. There is no MLP write, no layer norm, and no unembedding at the end. You cannot read off the next-token prediction from the canvas. A logit lens would be that reading, and this page does not do it.
How this relates to real models
Almost every later mech interp paper uses this stream as the shared notebook of the model. The Mathematical Framework is the source of the picture. When you leave the toy, you cache residual activations at each layer, decode them with the logit lens if you want a first impression, and then patch individual components to see which write mattered. Attribution graphs are a much larger version of "who wrote what into the stream for this prompt".
Read residual stream, MLP layers, and the logit lens, then the framework paper. The dimension sliders on this page are for intuition about overlap. They are not a result about any trained model.