Attention
What should the model focus on?
Pick a word and see how much "attention" it pays to every other word in the sentence — both as connecting lines and as a heatmap.
Attention doesn't simply mean "the AI understands what's important" — it's a mathematical mechanism that lets every token incorporate information from other tokens, with varying strengths.
In more detail
Attention lets each token gather information from other tokens. For every position, the model computes a set of weights over the positions it is allowed to look at (the weights are non-negative and add up to 1) and builds a new representation as a weighted mix of the information found there.
Consider "The animal didn't cross the street because it was too tired." To represent "it" usefully, the model needs information from "animal". Attention is the learned mechanism that makes this possible. In decoder-only models such as GPT, each token can look only at itself and the tokens before it (the causal mask), because the model generates text from left to right.
One caution: attention weights show where a head reads from, but they are not a complete explanation of why the model produced an output. Research has shown that a high attention weight does not always mean the token was decisive (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019). Also, a real model has many heads in many layers; one heatmap shows just one slice.
Simplification: the sentence and weights in the visualization are illustrative.