Embeddings
Tokens become vectors
Each token ID is looked up in a table and turned into a <b>vector</b> — a list of numbers. Words with related meanings tend to end up with nearby vectors.
The embedding space below is reduced to 2 dimensions for human eyes — real models use hundreds or thousands of dimensions.
Simplified visualization for teaching purposes — positions are illustrative, not real coordinates from a production model.
This is the classic illustration of what embeddings capture: relationships between words behave a bit like arrows you can add and subtract.
Illustrative geometry in a simplified 2D space — real embedding arithmetic happens across hundreds of dimensions and doesn't always land this cleanly.
In more detail
After tokenization, each token ID is used to look up one row in a large table called the embedding matrix. That row is a vector, a list of numbers (768 numbers in GPT-2 small, thousands in larger models), and it is the model's starting representation of that token. The values in the table are parameters learned during training, not designed by hand.
Training pushes tokens that are used in similar ways toward similar vectors, so directions in this space often reflect aspects of meaning or grammar. This is why "cat" and "dog" tend to end up closer to each other than to "carburetor". The classic illustration is the analogy king − man + woman ≈ queen, reported for word embeddings such as word2vec (Mikolov et al., 2013). Treat it as an illustration, not a law: in those experiments the words from the question are normally excluded from the candidate answers, results vary, and the input embeddings of modern language models do not behave so neatly.
Two important points:
- 1. The embedding of a token is the same wherever it appears. "Bank" starts from the same vector in "river bank" and in "bank account". It is the later layers, above all attention, that make the representation depend on context.
- 2. Real embedding spaces have hundreds to thousands of dimensions. The 2D picture is a projection made for human eyes, and it loses information.
Simplification: positions in the visualization are illustrative, not coordinates from a real model.