Key concepts in mechanistic interpretability

Mechanistic interpretability is a research field that tries to reverse-engineer neural networks into human-understandable algorithms, by identifying what internal components represent (features) and how they combine into computations (circuits), and testing these explanations with causal experiments on the model.

Residual stream

The residual stream is the path a token's information takes through a transformer. At each position there is one vector. It begins as the token's embedding, plus positional information. Every attention head and every MLP layer reads that vector and adds an update back onto it, instead of replacing it. The original information stays unless a later layer adds something that cancels it.

This picture comes from the 2021 paper "A Mathematical Framework for Transformer Circuits". It lets a researcher ask, at a chosen layer, what has been written so far and which component wrote it. The logit lens is a simple reading of the same stream: decode an intermediate vector as if it were the final one, and see which token it currently points toward.

Deeper note. A real stream has hundreds or thousands of dimensions. A component usually writes into only some directions, so two updates can share the stream without erasing each other. The MEJ7 residual-stream lab draws a toy version of this climb. The dimension counts and overlap numbers there are illustrative guesses, not measurements from a trained model.

Attention heads and the QK/OV circuits

An attention head moves information between token positions. The same framework splits that job in two. The QK circuit decides where the head looks: queries and keys produce a score for each earlier position, and softmax turns the scores into weights. The OV circuit decides what is copied: the value vector at a position is mapped into the residual stream, then mixed by those weights.

A layer has many heads. They read the same stream and write their updates back into it. Some heads in published studies mostly look at the previous token, some help copy a repeated pattern, and some do not have a clean human-readable role. A head is not a stored sentence. It is a small linear calculation plus a weighted sum.

Deeper note. The MEJ7 attention map and the value-mixing lab show the shape of this idea with a toy scoring function and three fixed arrows. They are not the QK or OV matrices of a trained model. In a real model those matrices are learned, and what a head copies has to be measured, not assumed from the picture.

MLP layers

An MLP layer, also called the feed-forward network, works at one token position. It does not look at other positions. It reads the residual stream at that position, expands the vector through a wider hidden layer with a non-linear activation, then writes a smaller update back. Attention is what moves information across the sentence. The MLP is what transforms it in place.

A widely used reading, from Geva et al. (2021), treats these layers as key-value memories: the first matrix can detect a pattern, and the second can add the associated direction into the stream. That is a useful picture, not a proof that every neuron stores one fact. Later work often finds the stored associations spread across many directions.

Deeper note. MEJ7 does not open a real MLP. The intermediate neural-network lab shows a tiny feed-forward network with teaching weights. A research tool such as a transcoder tries to rewrite one MLP as a sparse set of more interpretable features. That is an approximation, and it does not reconstruct the layer perfectly.

Features

A feature, in this field, is a direction in activation space that lines up with something a person can name: a concept, a language, a code pattern, a named entity, or a more abstract property. It is not automatically one neuron. A direction can be a combination of many neurons, and one neuron can take part in many directions.

Researchers look for features by reading activations, by training small probes, and by decomposing activations with sparse autoencoders. A probe can show that information is present. It does not, by itself, show that the model uses that information to produce the answer. A causal test, such as patching or steering, is what checks use.

Deeper note. Published work has reported features for specific concepts and for low-level patterns. The list is not a catalogue of everything a model knows. Features also split into finer ones when the dictionary grows, so a feature found at one dictionary size is not a final object.

Superposition

Superposition is the idea that a model stores more features than it has dimensions, by letting those features overlap. If the model rarely needs two features at the same time, it can pack them into shared dimensions and still recover each one often enough. Toy Models of Superposition (Elhage et al., 2022) studies this in small networks, where the geometry can actually be drawn.

The practical consequence is that a single neuron becomes a bad unit of explanation. It responds to a mixture. Looking at its top activating texts shows several unrelated themes, and none of them is the whole story. A direction that is sparse across many neurons can be cleaner than the neuron itself.

Deeper note. Superposition is an account of why interpretation is hard, backed by toy models and by the success of sparse decompositions. It is not a measurement that every production model uses the same packing. The MEJ7 embedding ring is a 3D drawing of "similar things sit nearer", not a map of superimposed features inside a real model.

Polysemanticity

Polysemanticity means one neuron responds to several things that are not the same concept. A neuron might activate for a particular word, for a piece of code, and for an unrelated topic, because superposition has folded those features onto the same unit. The opposite word, monosemanticity, describes a unit that lines up with one clean concept.

This is why sorting neurons by their highest activations often feels noisy. The neuron is real, and the activations are real, but the neuron is the wrong slice through the space. Sparse autoencoders are one attempt to cut the space along directions that activate rarely and, more often, mean one thing.

Deeper note. A neuron that looks monosemantic on a small dataset can still be polysemantic on a wider one. And a sparse feature that looks clean can still be wrong about what the model uses. Faithfulness, whether the explanation matches the computation, has to be tested. There is no agreed single score for that yet.

Circuits

A circuit is a subgraph of the network that carries out one identifiable computation. The pieces are usually attention heads and MLP layers, connected through the residual stream. The claim is not "these parts are near each other". The claim is "this small set is sufficient for this behaviour, and removing it breaks that behaviour".

The classic documented example is indirect object identification in GPT-2 small (Wang et al., 2022): a set of heads that completes "When Mary and John went to the store, John gave a drink to ___" with "Mary". Induction heads are another circuit, a pair that continues a repeated pattern. Both were studied with ablations and with activation patching, not by looking at a diagram alone.

Deeper note. Most fully traced circuits are in small models or on narrow tasks. A circuit that explains one prompt family can fail on a nearby family. Automated search, such as ACDC (Conmy et al., 2023), helps find candidates. It does not replace the check that the story is faithful.

Induction heads

An induction head is an attention pattern that completes a repetition. If the context contains "A B" and later an "A", the head helps predict "B". In the published account, this is done by a pair of heads: one marks that a token appeared earlier, and a later head copies what followed it. The pattern is written "A B ... A → B".

Olsson et al. (2022) report that these heads appear fairly suddenly during training and that they are tied to in-context learning, the ability to use a pattern shown inside the current prompt. That is evidence from the models they studied. It is not a claim that every in-context ability, in every model, is only induction.

Deeper note. You cannot see an induction head in the MEJ7 attention toy. That toy scores tokens with a fixed teaching rule, so the bright cells are not a learned "copy the previous continuation" circuit. To study the real thing, people load a small model in TransformerLens and look for the previous-token and copy pattern on repeated text.

Sparse autoencoders

A sparse autoencoder, or SAE, is a small extra network trained on a model's activations. It tries to rebuild each activation as a sum of a few directions from a large dictionary. The sparsity penalty pushes most dictionary entries to stay off, so the ones that turn on are candidates for features. This is dictionary learning applied to the residual stream or to an MLP.

Bricken et al. (2023) showed interpretable features in a small transformer. Later papers applied the same idea at larger scale, including Claude 3 Sonnet (Templeton et al., 2024) and the open Gemma Scope suite (Lieberum et al., 2024). Independent work, including Cunningham et al. (2023) and Gao et al. (2024), studies how to train and measure these dictionaries.

Deeper note. An SAE does not reconstruct activations perfectly, and the features change with the dictionary size, the sparsity setting, and the training data. A beautiful feature dashboard is not yet proof that the model uses that feature. Patching, ablation, or steering is still needed. MEJ7 does not run an SAE. Neuronpedia is one public place where trained features can be browsed.

Activation patching

Activation patching, also called causal tracing in some papers, tests which internal value actually matters. You run the model on two inputs, a clean one and a corrupted one. Then you copy one activation, at one component and one position, from the clean run into the corrupted run, and you measure how far the output moves back. If the answer returns, that activation carried something the model used.

The method is stronger than a probe because it changes the computation and watches the result. It is also easy to over-read. A component can score high because it carries a copy of information that another component also carries, or because the corruption was narrow. Patching is usually repeated across positions and layers, and then checked with an ablation.

Deeper note. Meng et al. (2022) used causal tracing to locate where some factual associations are used, and then edited them. That result is about the models and the facts they tested. It does not mean every fact sits in one editable cell. None of the MEJ7 labs perform patching. They do not have a second run to copy from.

Logit lens

The logit lens reads an intermediate residual-stream vector with the model's own unembedding, the matrix that turns the final vector into a score for every token. You ask: if the model had to answer from this layer, which token would be in the lead? It is a cheap view of how the prediction forms, and it needs no extra trained probe.

Early layers are often vague, and later layers sharpen. Sometimes a token becomes likely in the middle of the network and then gets revised. That is a clue about when information arrived, not a full circuit. The lens can also mislead, because the unembedding was trained to read the final vector, not every intermediate one. A tuned variant, the tuned lens, exists for that reason. MEJ7 does not include it.

Deeper note. Use the lens as a first look, then check with patching. A token that looks likely at layer 6 may be ignored by the real path that produces the final answer. The MEJ7 residual animation shows a toy vector climbing layers. It does not decode that vector into the vocabulary of a real model.

Steering

Steering adds or subtracts a direction in the activations while the model runs, then watches whether the behaviour changes. If a direction is supposed to mean "this is a refusal" or "this is a particular topic", adding it should push the text that way, and subtracting it should push the other way. It is a causal test of a feature, not only a description of one.

The direction might come from a probe, from the difference between two sets of prompts, or from a sparse-autoencoder feature. Representation engineering (Zou et al., 2023) is a related top-down version of the same idea: find a direction for a concept and use it to read or to steer. Success on a demo prompt is not success on every prompt.

Deeper note. Steering can change style, topic, or a measured behaviour and still leave the rest of the circuit untouched. A failed steer does not always mean the direction is meaningless. The scale can be wrong, or the direction can matter only at one layer. MEJ7's sliders change toy scores. They do not add a learned direction to a language model.

Grokking

Grokking is a training surprise. A small network first memorizes its training set, looks bad on new examples for a long time, and then, much later, generalizes. The weights keep changing after the training loss is already low. Nanda et al. (2023) studied one clean case: a tiny transformer trained on modular addition. They reverse-engineered the late solution into an algorithm that uses Fourier features, sums of waves, rather than a lookup table.

The paper is important because the algorithm can be stated and then checked. It is also narrow. It does not say that a language model "groks language" on a schedule, and it does not give a date when a large model will suddenly understand a task. Sudden jumps during training are a separate observation. Induction heads are one reported example of a capability that appears sharply.

Deeper note. If you repeat the modular-addition experiment, use the published setup and expect to spend real time on the analysis, not an afternoon on a toy slider. MEJ7 has no grokking animation, because a honest one would have to train the small network and show the phase change, not a drawn curve.

Author: mej7.com, an independent educational project. Last reviewed: 6 November 2026.