mej7.comHow AI Works
EN SHQIP
Beginner Intermediate Advanced Mechanistic Interpretability

Next-Token Generation

Lesson 16 of 28 · Intermediate level · interactive visualization (simplified teaching model)

AUTOREGRESSIVE GENERATION

Next-token generation simulator

The model generates one token at a time, adds it to the context, and repeats. Click a candidate token to add it.

CURRENT CONTEXT
Context↓Tokenization↓Model↓Logits↓Softmax↓Choose Token↓Add To Context↻

In more detail

A language model produces text one token at a time:

  1. 1. Take the current text (the context) and run it through the model.
  2. 2. Get a probability distribution over the next token.
  3. 3. Choose a token (by sampling or by taking the most probable one).
  4. 4. Append it to the text and repeat.

Generation stops when the model produces a special end-of-sequence token or reaches a length limit. Because every token depends on the ones before it, early choices influence everything that follows, and because choosing usually involves randomness, the same prompt can give different answers.

This is also why longer answers take longer: each new token needs another pass through the model. In practice, implementations store the keys and values of earlier tokens (the KV cache, see the Advanced track) so they do not have to be recomputed each time.

Simplification: the simulator lets you pick from a few candidate tokens. A real model computes a probability for every token in its vocabulary.