Context Window
How much the model can "see" at once
The model can only see a limited number of tokens at a time — the context window.
Context windows are measured in tokens and vary by model.
In more detail
The context window is the maximum number of tokens the model can take into account at once. It includes your prompt, the earlier conversation, and the model's own answer so far. GPT-2 had a window of 1,024 tokens; modern models range from thousands to hundreds of thousands of tokens or more, depending on the model.
Limits exist because attention compares each token with all the earlier ones, so its cost grows with the square of the length in a straightforward implementation, and because the stored keys and values take memory.
What happens when the text does not fit depends on the application. It may refuse, cut off the oldest messages, or summarize them. Some architectures use sliding-window attention, where each token can see only a recent window.
Two further points:
- Fitting into the window does not guarantee that the model uses all the information equally well. Studies have found that models can be worse at using information placed in the middle of a long context (Liu et al., 2023).
- The model does not remember earlier, separate conversations unless the application stores information and supplies it again in the context.