What is mechanistic interpretability?

Mechanistic interpretability is a research field that tries to reverse-engineer neural networks into human-understandable algorithms, by identifying what internal components represent (features) and how they combine into computations (circuits), and testing these explanations with causal experiments on the model.

Mechanistic interpretability (often shortened to "mech interp") is a branch of AI research that tries to understand how a neural network produces its outputs by studying its internal parts, not just its inputs and outputs. The goal is ambitious: take a trained model, which is a huge collection of numbers, and turn it into an explanation a person can read, such as "this group of components detects the previous occurrence of a word and copies what came after it."

Why it exists

Modern language models are trained, not programmed. Nobody writes the rules they follow. We choose an architecture, give it data and an objective, and training adjusts billions of numbers until the model works. The result performs impressive tasks, but nobody can open it and read off how. Mechanistic interpretability tries to change that. The motivation is partly scientific curiosity (what algorithms does gradient descent discover?) and partly safety: if we can see what a model is actually computing, we may be able to detect deception, bias, dangerous capabilities or simple bugs before they cause harm.

What it studies

Researchers in this field ask questions at several levels:

• Features: what does a direction or a unit inside the model represent? Examples reported in published work include features for specific concepts, languages, code patterns or named entities.
• Circuits: which components connect to form a computation? A circuit is a subgraph of the network, built from attention heads and MLP layers, that performs an identifiable task.
• Algorithms: can the whole computation be described as a procedure, like modular addition done with trigonometric identities, which researchers found in small networks trained on arithmetic?
• Training dynamics: how do these circuits form as the model learns, and why do capabilities sometimes appear suddenly?

How researchers do it (the core methods)

1. Reading internal activations. Record what happens inside the model on a given input. Simple tools like the logit lens look at what the model "would predict" at intermediate layers.
2. Probing. Train a small classifier on internal activations to test whether some information is present. Probing shows correlation, not that the model uses it.
3. Ablation. Remove or zero a component and measure what breaks. This tests whether it matters.
4. Activation patching (causal tracing). Run the model on two inputs, copy an internal activation from one run into the other, and see how the output changes. This identifies which components carry causally relevant information.
5. Sparse autoencoders (SAEs) and dictionary learning. Because single neurons often respond to many unrelated things (polysemanticity), researchers train an auxiliary network to find a larger set of more interpretable directions, called features.
6. Circuit tracing and attribution. Follow how features influence each other layer by layer to produce an answer for a specific prompt.
7. Steering. Add or subtract a direction in the model's activations to change its behavior, which tests whether the direction really does what is claimed.

The transformer picture researchers use

Most work views a transformer through the residual stream: a running vector for each token position that every layer reads from and writes to. Attention heads move information between positions. MLP layers process information at one position. Each attention head can be split into a "where to look" part (the QK circuit) and a "what to copy" part (the OV circuit). This view comes from the 2021 paper "A Mathematical Framework for Transformer Circuits" and is the foundation for most later work.

Famous findings (simplified)

• Induction heads: a pair of attention heads that lets a model continue patterns like "A B ... A → B". They appear suddenly during training and are linked to in-context learning.
• Indirect object identification circuit in GPT-2 small: a documented circuit of attention heads that lets the model complete "When Mary and John went to the store, John gave a drink to ___" with "Mary".
• Grokking and modular addition: small networks trained on arithmetic were reverse-engineered into an algorithm based on Fourier features.
• Superposition: models appear to store more features than they have dimensions by overlapping them, which is one reason neurons are hard to interpret.
• Large-scale features: sparse autoencoders have been applied to production-scale language models, extracting millions of features, and attribution graphs have been used to study multi-step reasoning inside a model.

What it cannot do yet (honest limits)

• Explanations can be partial, and an explanation that sounds right may not be faithful to what the model really does.
• Sparse autoencoders do not reconstruct activations perfectly, and the features they find depend on training choices.
• Most detailed circuit analyses are on small models or on narrow tasks. Scaling to full frontier models is an open problem.
• There is no consensus yet on how to measure whether an interpretation is correct.
• The field is young, and some earlier conclusions have been revised.

How MEJ7 fits in

MEJ7's mechanistic interpretability labs are toy models built for intuition: the attention maps, value mixing, residual stream and 3D embeddings you see here are illustrative, not real model weights. To do real mechanistic interpretability, use the tools and courses on our learning path and tools pages.

Author: mej7.com, an independent educational project. Last reviewed: 6 November 2026.