Look inside attention, values, and the residual stream

What is mechanistic interpretability?

Mechanistic interpretability is a research field that tries to reverse-engineer neural networks into human-understandable algorithms, by identifying what internal components represent (features) and how they combine into computations (circuits), and testing these explanations with causal experiments on the model.

Mechanistic interpretability (often shortened to "mech interp") is a branch of AI research that tries to understand how a neural network produces its outputs by studying its internal parts, not just its inputs and outputs. The goal is ambitious: take a trained model, which is a huge collection of numbers, and turn it into an explanation a person can read, such as "this group of components detects the previous occurrence of a word and copies what came after it."

Why it exists

Modern language models are trained, not programmed. Nobody writes the rules they follow. We choose an architecture, give it data and an objective, and training adjusts billions of numbers until the model works. The result performs impressive tasks, but nobody can open it and read off how. Mechanistic interpretability tries to change that. The motivation is partly scientific curiosity (what algorithms does gradient descent discover?) and partly safety: if we can see what a model is actually computing, we may be able to detect deception, bias, dangerous capabilities or simple bugs before they cause harm.

What it studies

Researchers in this field ask questions at several levels:

• Features: what does a direction or a unit inside the model represent? Examples reported in published work include features for specific concepts, languages, code patterns or named entities.
• Circuits: which components connect to form a computation? A circuit is a subgraph of the network, built from attention heads and MLP layers, that performs an identifiable task.
• Algorithms: can the whole computation be described as a procedure, like modular addition done with trigonometric identities, which researchers found in small networks trained on arithmetic?
• Training dynamics: how do these circuits form as the model learns, and why do capabilities sometimes appear suddenly?

How researchers do it (the core methods)

1. Reading internal activations. Record what happens inside the model on a given input. Simple tools like the logit lens look at what the model "would predict" at intermediate layers.
2. Probing. Train a small classifier on internal activations to test whether some information is present. Probing shows correlation, not that the model uses it.
3. Ablation. Remove or zero a component and measure what breaks. This tests whether it matters.
4. Activation patching (causal tracing). Run the model on two inputs, copy an internal activation from one run into the other, and see how the output changes. This identifies which components carry causally relevant information.
5. Sparse autoencoders (SAEs) and dictionary learning. Because single neurons often respond to many unrelated things (polysemanticity), researchers train an auxiliary network to find a larger set of more interpretable directions, called features.
6. Circuit tracing and attribution. Follow how features influence each other layer by layer to produce an answer for a specific prompt.
7. Steering. Add or subtract a direction in the model's activations to change its behavior, which tests whether the direction really does what is claimed.

The transformer picture researchers use

Most work views a transformer through the residual stream: a running vector for each token position that every layer reads from and writes to. Attention heads move information between positions. MLP layers process information at one position. Each attention head can be split into a "where to look" part (the QK circuit) and a "what to copy" part (the OV circuit). This view comes from the 2021 paper "A Mathematical Framework for Transformer Circuits" and is the foundation for most later work.

Famous findings (simplified)

• Induction heads: a pair of attention heads that lets a model continue patterns like "A B ... A → B". They appear suddenly during training and are linked to in-context learning.
• Indirect object identification circuit in GPT-2 small: a documented circuit of attention heads that lets the model complete "When Mary and John went to the store, John gave a drink to ___" with "Mary".
• Grokking and modular addition: small networks trained on arithmetic were reverse-engineered into an algorithm based on Fourier features.
• Superposition: models appear to store more features than they have dimensions by overlapping them, which is one reason neurons are hard to interpret.
• Large-scale features: sparse autoencoders have been applied to production-scale language models, extracting millions of features, and attribution graphs have been used to study multi-step reasoning inside a model.

What it cannot do yet (honest limits)

• Explanations can be partial, and an explanation that sounds right may not be faithful to what the model really does.
• Sparse autoencoders do not reconstruct activations perfectly, and the features they find depend on training choices.
• Most detailed circuit analyses are on small models or on narrow tasks. Scaling to full frontier models is an open problem.
• There is no consensus yet on how to measure whether an interpretation is correct.
• The field is young, and some earlier conclusions have been revised.

How MEJ7 fits in

MEJ7's mechanistic interpretability labs are toy models built for intuition: the attention maps, value mixing, residual stream and 3D embeddings you see here are illustrative, not real model weights. To do real mechanistic interpretability, use the tools and courses on our learning path and tools pages.

Key concepts

Residual stream

The residual stream is the path a token's information takes through a transformer. At each position there is one vector. It begins as the token's embedding, plus positional information. Every attention head and every MLP layer reads that vector and adds an update back onto it, instead of replacing it. The original information stays unless a later layer adds something that cancels it.

This picture comes from the 2021 paper "A Mathematical Framework for Transformer Circuits". It lets a researcher ask, at a chosen layer, what has been written so far and which component wrote it. The logit lens is a simple reading of the same stream: decode an intermediate vector as if it were the final one, and see which token it currently points toward.

Deeper note. A real stream has hundreds or thousands of dimensions. A component usually writes into only some directions, so two updates can share the stream without erasing each other. The MEJ7 residual-stream lab draws a toy version of this climb. The dimension counts and overlap numbers there are illustrative guesses, not measurements from a trained model.

Attention heads and the QK/OV circuits

An attention head moves information between token positions. The same framework splits that job in two. The QK circuit decides where the head looks: queries and keys produce a score for each earlier position, and softmax turns the scores into weights. The OV circuit decides what is copied: the value vector at a position is mapped into the residual stream, then mixed by those weights.

A layer has many heads. They read the same stream and write their updates back into it. Some heads in published studies mostly look at the previous token, some help copy a repeated pattern, and some do not have a clean human-readable role. A head is not a stored sentence. It is a small linear calculation plus a weighted sum.

Deeper note. The MEJ7 attention map and the value-mixing lab show the shape of this idea with a toy scoring function and three fixed arrows. They are not the QK or OV matrices of a trained model. In a real model those matrices are learned, and what a head copies has to be measured, not assumed from the picture.

MLP layers

An MLP layer, also called the feed-forward network, works at one token position. It does not look at other positions. It reads the residual stream at that position, expands the vector through a wider hidden layer with a non-linear activation, then writes a smaller update back. Attention is what moves information across the sentence. The MLP is what transforms it in place.

A widely used reading, from Geva et al. (2021), treats these layers as key-value memories: the first matrix can detect a pattern, and the second can add the associated direction into the stream. That is a useful picture, not a proof that every neuron stores one fact. Later work often finds the stored associations spread across many directions.

Deeper note. MEJ7 does not open a real MLP. The intermediate neural-network lab shows a tiny feed-forward network with teaching weights. A research tool such as a transcoder tries to rewrite one MLP as a sparse set of more interpretable features. That is an approximation, and it does not reconstruct the layer perfectly.

Features

A feature, in this field, is a direction in activation space that lines up with something a person can name: a concept, a language, a code pattern, a named entity, or a more abstract property. It is not automatically one neuron. A direction can be a combination of many neurons, and one neuron can take part in many directions.

Researchers look for features by reading activations, by training small probes, and by decomposing activations with sparse autoencoders. A probe can show that information is present. It does not, by itself, show that the model uses that information to produce the answer. A causal test, such as patching or steering, is what checks use.

Deeper note. Published work has reported features for specific concepts and for low-level patterns. The list is not a catalogue of everything a model knows. Features also split into finer ones when the dictionary grows, so a feature found at one dictionary size is not a final object.

Superposition

Superposition is the idea that a model stores more features than it has dimensions, by letting those features overlap. If the model rarely needs two features at the same time, it can pack them into shared dimensions and still recover each one often enough. Toy Models of Superposition (Elhage et al., 2022) studies this in small networks, where the geometry can actually be drawn.

The practical consequence is that a single neuron becomes a bad unit of explanation. It responds to a mixture. Looking at its top activating texts shows several unrelated themes, and none of them is the whole story. A direction that is sparse across many neurons can be cleaner than the neuron itself.

Deeper note. Superposition is an account of why interpretation is hard, backed by toy models and by the success of sparse decompositions. It is not a measurement that every production model uses the same packing. The MEJ7 embedding ring is a 3D drawing of "similar things sit nearer", not a map of superimposed features inside a real model.

Polysemanticity

Polysemanticity means one neuron responds to several things that are not the same concept. A neuron might activate for a particular word, for a piece of code, and for an unrelated topic, because superposition has folded those features onto the same unit. The opposite word, monosemanticity, describes a unit that lines up with one clean concept.

This is why sorting neurons by their highest activations often feels noisy. The neuron is real, and the activations are real, but the neuron is the wrong slice through the space. Sparse autoencoders are one attempt to cut the space along directions that activate rarely and, more often, mean one thing.

Deeper note. A neuron that looks monosemantic on a small dataset can still be polysemantic on a wider one. And a sparse feature that looks clean can still be wrong about what the model uses. Faithfulness, whether the explanation matches the computation, has to be tested. There is no agreed single score for that yet.

Circuits

A circuit is a subgraph of the network that carries out one identifiable computation. The pieces are usually attention heads and MLP layers, connected through the residual stream. The claim is not "these parts are near each other". The claim is "this small set is sufficient for this behaviour, and removing it breaks that behaviour".

The classic documented example is indirect object identification in GPT-2 small (Wang et al., 2022): a set of heads that completes "When Mary and John went to the store, John gave a drink to ___" with "Mary". Induction heads are another circuit, a pair that continues a repeated pattern. Both were studied with ablations and with activation patching, not by looking at a diagram alone.

Deeper note. Most fully traced circuits are in small models or on narrow tasks. A circuit that explains one prompt family can fail on a nearby family. Automated search, such as ACDC (Conmy et al., 2023), helps find candidates. It does not replace the check that the story is faithful.

Induction heads

An induction head is an attention pattern that completes a repetition. If the context contains "A B" and later an "A", the head helps predict "B". In the published account, this is done by a pair of heads: one marks that a token appeared earlier, and a later head copies what followed it. The pattern is written "A B ... A → B".

Olsson et al. (2022) report that these heads appear fairly suddenly during training and that they are tied to in-context learning, the ability to use a pattern shown inside the current prompt. That is evidence from the models they studied. It is not a claim that every in-context ability, in every model, is only induction.

Deeper note. You cannot see an induction head in the MEJ7 attention toy. That toy scores tokens with a fixed teaching rule, so the bright cells are not a learned "copy the previous continuation" circuit. To study the real thing, people load a small model in TransformerLens and look for the previous-token and copy pattern on repeated text.

Sparse autoencoders

A sparse autoencoder, or SAE, is a small extra network trained on a model's activations. It tries to rebuild each activation as a sum of a few directions from a large dictionary. The sparsity penalty pushes most dictionary entries to stay off, so the ones that turn on are candidates for features. This is dictionary learning applied to the residual stream or to an MLP.

Bricken et al. (2023) showed interpretable features in a small transformer. Later papers applied the same idea at larger scale, including Claude 3 Sonnet (Templeton et al., 2024) and the open Gemma Scope suite (Lieberum et al., 2024). Independent work, including Cunningham et al. (2023) and Gao et al. (2024), studies how to train and measure these dictionaries.

Deeper note. An SAE does not reconstruct activations perfectly, and the features change with the dictionary size, the sparsity setting, and the training data. A beautiful feature dashboard is not yet proof that the model uses that feature. Patching, ablation, or steering is still needed. MEJ7 does not run an SAE. Neuronpedia is one public place where trained features can be browsed.

Activation patching

Activation patching, also called causal tracing in some papers, tests which internal value actually matters. You run the model on two inputs, a clean one and a corrupted one. Then you copy one activation, at one component and one position, from the clean run into the corrupted run, and you measure how far the output moves back. If the answer returns, that activation carried something the model used.

The method is stronger than a probe because it changes the computation and watches the result. It is also easy to over-read. A component can score high because it carries a copy of information that another component also carries, or because the corruption was narrow. Patching is usually repeated across positions and layers, and then checked with an ablation.

Deeper note. Meng et al. (2022) used causal tracing to locate where some factual associations are used, and then edited them. That result is about the models and the facts they tested. It does not mean every fact sits in one editable cell. None of the MEJ7 labs perform patching. They do not have a second run to copy from.

Logit lens

The logit lens reads an intermediate residual-stream vector with the model's own unembedding, the matrix that turns the final vector into a score for every token. You ask: if the model had to answer from this layer, which token would be in the lead? It is a cheap view of how the prediction forms, and it needs no extra trained probe.

Early layers are often vague, and later layers sharpen. Sometimes a token becomes likely in the middle of the network and then gets revised. That is a clue about when information arrived, not a full circuit. The lens can also mislead, because the unembedding was trained to read the final vector, not every intermediate one. A tuned variant, the tuned lens, exists for that reason. MEJ7 does not include it.

Deeper note. Use the lens as a first look, then check with patching. A token that looks likely at layer 6 may be ignored by the real path that produces the final answer. The MEJ7 residual animation shows a toy vector climbing layers. It does not decode that vector into the vocabulary of a real model.

Steering

Steering adds or subtracts a direction in the activations while the model runs, then watches whether the behaviour changes. If a direction is supposed to mean "this is a refusal" or "this is a particular topic", adding it should push the text that way, and subtracting it should push the other way. It is a causal test of a feature, not only a description of one.

The direction might come from a probe, from the difference between two sets of prompts, or from a sparse-autoencoder feature. Representation engineering (Zou et al., 2023) is a related top-down version of the same idea: find a direction for a concept and use it to read or to steer. Success on a demo prompt is not success on every prompt.

Deeper note. Steering can change style, topic, or a measured behaviour and still leave the rest of the circuit untouched. A failed steer does not always mean the direction is meaningless. The scale can be wrong, or the direction can matter only at one layer. MEJ7's sliders change toy scores. They do not add a learned direction to a language model.

Grokking

Grokking is a training surprise. A small network first memorizes its training set, looks bad on new examples for a long time, and then, much later, generalizes. The weights keep changing after the training loss is already low. Nanda et al. (2023) studied one clean case: a tiny transformer trained on modular addition. They reverse-engineered the late solution into an algorithm that uses Fourier features, sums of waves, rather than a lookup table.

The paper is important because the algorithm can be stated and then checked. It is also narrow. It does not say that a language model "groks language" on a schedule, and it does not give a date when a large model will suddenly understand a task. Sudden jumps during training are a separate observation. Induction heads are one reported example of a capability that appears sharply.

Deeper note. If you repeat the modular-addition experiment, use the published setup and expect to spend real time on the analysis, not an afternoon on a toy slider. MEJ7 has no grokking animation, because a honest one would have to train the small network and show the phase change, not a drawn curve.

Recommended reading

Reading order is MEJ7's editorial recommendation, last reviewed 6 November 2026.

Step 0: A gentle start (read or watch first)

1. Zoom In: An Introduction to Circuits — Olah et al., 2020. Distill. The original statement of the circuits agenda, with features, circuits and universality.
2. Neural Networks — 3Blue1Brown. Video series. Visual foundation for neural networks and transformers.
3. A Comprehensive Mechanistic Interpretability Explainer & Glossary — Neel Nanda. neelnanda.io. A long practical glossary by a leading researcher.

Step 1: Transformer foundations

4. A Mathematical Framework for Transformer Circuits — Elhage et al., 2021. Anthropic, Transformer Circuits. Introduces the residual-stream view and QK/OV circuits. The single most important starting paper.
5. Attention Is All You Need — Vaswani et al., 2017. NeurIPS. The original transformer paper.
6. Let's build GPT — Andrej Karpathy. Video. Build a small GPT from scratch to understand every component.

Step 2: Circuits and mechanisms

7. In-context Learning and Induction Heads — Olsson et al., 2022. Anthropic, Transformer Circuits. Evidence that induction heads drive in-context learning.
8. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small — Wang et al., 2022. arXiv. A full circuit analysis in a real model.
9. Progress measures for grokking via mechanistic interpretability — Nanda et al., 2023. ICLR. Reverse-engineers a small network's modular addition algorithm.
10. Locating and Editing Factual Associations in GPT — Meng et al., 2022. NeurIPS. Causal tracing of where facts are stored and how to edit them.
11. Transformer Feed-Forward Layers Are Key-Value Memories — Geva et al., 2021. EMNLP. Views MLP layers as memories of patterns.
12. Towards Automated Circuit Discovery for Mechanistic Interpretability — Conmy et al., 2023. NeurIPS. Automating circuit finding (ACDC).

Step 3: Superposition and features

13. Toy Models of Superposition — Elhage et al., 2022. Anthropic, Transformer Circuits. Why neurons are polysemantic and how superposition works in small models.
14. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Bricken et al., 2023. Anthropic, Transformer Circuits. Sparse autoencoders find interpretable features in a small transformer.
15. Sparse Autoencoders Find Highly Interpretable Features in Language Models — Cunningham et al., 2023. arXiv. Independent SAE results.
16. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Templeton et al., 2024. Anthropic, Transformer Circuits. SAEs applied to a production-scale model.
17. Scaling and evaluating sparse autoencoders — Gao et al., 2024. OpenAI. Methods and metrics for large SAEs.
18. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 — Lieberum et al., 2024. Google DeepMind. Open SAE suite for researchers.

Step 4: Overview and large scale

19. Circuit Tracing: Revealing Computational Graphs in Language Models and On the Biology of a Large Language Model — Anthropic, 2025. Transformer Circuits. Attribution graphs for studying multi-step computation in a large model.
20. Mechanistic Interpretability for AI Safety — A Review — Bereska and Gavves, 2024. arXiv. A broad survey of the field and its safety relevance.
21. A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models — Rai et al., 2024. arXiv. A practitioner-oriented review.
22. Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al., 2023. arXiv. A related top-down approach (reading and steering representations).
23. 200 Concrete Open Problems in Mechanistic Interpretability — Neel Nanda. Alignment Forum. A source of research project ideas.

Blogs and places to read

• Transformer Circuits Thread (transformer-circuits.pub): Anthropic's papers.
• Distill (distill.pub, archive): visual articles.
• Alignment Forum / LessWrong: discussion and posts by researchers.
• neelnanda.io: guides and videos.

Learning path

Main courses

ARENA (Alignment Research Engineer Accelerator)

arena.education. A technical training program (originally run in person in London) whose full curriculum is published openly. It includes chapters on transformers from scratch, introduction to mechanistic interpretability, superposition and sparse autoencoders, circuits such as indirect object identification, and more, with coding exercises and solutions. Created by Callum McDougall and collaborators. It is one of the best structured free routes into the field. Check the website for current programs and materials.

Who it is for: people who know basic Python and PyTorch and want to code it themselves.

Neel Nanda — starting materials

neelnanda.io. Practical guides on how to get started in mechanistic interpretability, a long glossary, video walkthroughs of papers and coding, and lists of open problems. Free.

Karpathy — "Neural Networks: Zero to Hero"

karpathy.ai/zero-to-hero, including Let's build GPT. Builds from backpropagation to a GPT. Ideal prerequisite.

3Blue1Brown — Neural Networks

3blue1brown.com/topics/neural-networks. Best visual intuition for beginners.

MATS (ML Alignment & Theory Scholars)

matsprogram.org. A selective research program that matches participants with mentors, including mechanistic interpretability researchers. For people ready to do real research. Check current applications.

BlueDot Impact — AI Safety courses

bluedot.org. Introductory courses on AI safety, useful context for why interpretability matters. Not specific to mech interp.

Step by step (with time as a guide)

These are rough estimates, not promises. The pace depends on your experience.

Tools

MEJ7's own labs are toys. These are the libraries and sites people use on real models.

Links were checked on 6 November 2026. Software moves; confirm that a project is still maintained before you rely on it.

Glossary

Longer explanations are on the key concepts page. Neel Nanda's glossary is the long field reference.

Frequently asked questions

What is mechanistic interpretability in simple words?

It is the study of what happens inside an AI model, trying to find the actual internal "wiring" that produces its answers, rather than treating it as a black box.

How is it different from explainable AI (XAI)?

Many XAI methods explain which inputs mattered (such as saliency maps). Mechanistic interpretability aims to explain the internal computation itself.

Do I need to know math?

You need basic linear algebra (vectors, matrices, dot products) and some probability. Advanced mathematics helps later but is not required to start.

Do I need to code?

Yes, practical work uses Python and PyTorch. You can learn concepts first with MEJ7's visual labs.

Where should a beginner start?

Start with MEJ7's beginner and intermediate labs, then Karpathy's GPT video, then the ARENA introduction and "A Mathematical Framework for Transformer Circuits".

What are the best mechanistic interpretability papers?

See our reading list, which starts with "A Mathematical Framework for Transformer Circuits" and covers induction heads, superposition and sparse autoencoders.

Is MEJ7's mech interp lab a real model?

No. It is a toy illustration for intuition. Real analysis uses libraries like TransformerLens on real models.

Is this field useful for a career?

It is an active research area at AI labs and academic groups, but opportunities are competitive and change quickly. Check current job and program listings rather than relying on general claims.

Author: mej7.com, an independent educational project. Last reviewed: 6 November 2026.