Logits & Softmax
How does AI choose the next word?
The model produces a number ("logit") for every possible token, then <b>softmax</b> turns them into probabilities.
Low temperature → more concentrated distribution. High temperature → more spread out / random. This doesn't make the AI "smarter" or "dumber" — it just changes the randomness of the choice.
In more detail
The final vector at the last position is multiplied by the unembedding matrix. The result is one score, called a logit, for every token in the vocabulary (50,257 in GPT-2). Logits can be any real numbers; they are not probabilities.
Softmax turns them into probabilities: p_i = e^(z_i) / Σ_j e^(z_j). All results are positive and add up to 1. Only differences between logits matter: adding the same number to all of them changes nothing.
Temperature (T) divides the logits before softmax. With T below 1 the distribution becomes sharper, so likely tokens become even more likely. With T above 1 it becomes flatter and less likely tokens get more chance. As T approaches 0, the choice approaches always taking the top token, and implementations treat T = 0 as "always pick the highest". Temperature does not change what the model knows; it changes how it samples from its own distribution. Very high temperatures often produce incoherent text.
Common sampling strategies:
- Greedy: always take the most probable token.
- Top-k: keep only the k most probable tokens, renormalize, and sample.
- Top-p (nucleus) sampling: keep the smallest set of tokens whose probabilities add up to at least p, and sample from it (Holtzman et al., 2019).
Chat products usually choose such settings for you.