3D Embeddings
Words live as points in space — rotate and see
Every token becomes a vector of numbers — a point in a high-dimensional space where similar meanings sit close together. Here's a 3D slice of that idea, spinning slowly.
Positions here are illustrative, not computed from a real embedding model — but the core idea is exactly this: related words (animals, actions, places) cluster together in the space a transformer actually reasons in.
Toy model for intuition, not real model weights.
What you are seeing
A ring of words turns in three dimensions. Animal words sit nearer other animal words, actions nearer other actions, places nearer other places. The picture is the everyday claim about embeddings: a token is a point, and related points were pulled together during training. You can rotate the ring and see that "near" depends on the angle. Two words that look close from the front can separate when you turn the cloud.
Mechanistic interpretability inherits this geometry and then asks a sharper question. A feature is not the point. A feature is a direction through the space. Many directions can be packed into the same dimensions, which is superposition. The ring is only the first idea, that distance means something. It is not a map of those directions.
What is simplified here
The coordinates were chosen so the clusters are obvious to a person watching a screen. They were not taken from a model's embedding matrix, and they were not produced by a dimensionality-reduction algorithm such as PCA. A real embedding has hundreds or thousands of dimensions. Any 3D view throws most of them away, so neighbourhoods in the picture are not neighbourhoods in the model. The disclaimer on the canvas says the positions are illustrative. That is the whole status of the figure.
The ring also freezes one vector per word. In a transformer the residual stream at a position starts near the embedding and then moves as layers write into it. The point you would want for a feature is often that later vector, not the input embedding. This page does not show the movement.
How this relates to real models
Word2vec made the "king minus man plus woman" picture famous, and the intermediate embeddings lesson on MEJ7 already treats that as an illustration with known caveats. Sparse autoencoders are the tool this field uses when the raw dimensions are too mixed to name. They look for directions that fire rarely and line up with a concept a person recognizes. Those directions are not visible by spinning this ring.
Read features, superposition, and sparse autoencoders, then Toy Models of Superposition and Towards Monosemanticity. To browse features from real models, use Neuronpedia. Use this ring to remember that the model stores tokens as geometry. Do not use it as evidence of what any particular model clusters.