Mechanistic interpretability reading list

Reading order is MEJ7's editorial recommendation, last reviewed 6 November 2026.

Step 0: A gentle start (read or watch first)

Zoom In: An Introduction to Circuits — Olah et al., 2020. Distill. The original statement of the circuits agenda, with features, circuits and universality.
Neural Networks — 3Blue1Brown. Video series. Visual foundation for neural networks and transformers.
A Comprehensive Mechanistic Interpretability Explainer & Glossary — Neel Nanda. neelnanda.io. A long practical glossary by a leading researcher.

A Mathematical Framework for Transformer Circuits — Elhage et al., 2021. Anthropic, Transformer Circuits. Introduces the residual-stream view and QK/OV circuits. The single most important starting paper.
Attention Is All You Need — Vaswani et al., 2017. NeurIPS. The original transformer paper.
Let's build GPT — Andrej Karpathy. Video. Build a small GPT from scratch to understand every component.

In-context Learning and Induction Heads — Olsson et al., 2022. Anthropic, Transformer Circuits. Evidence that induction heads drive in-context learning.
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small — Wang et al., 2022. arXiv. A full circuit analysis in a real model.
Progress measures for grokking via mechanistic interpretability — Nanda et al., 2023. ICLR. Reverse-engineers a small network's modular addition algorithm.
Locating and Editing Factual Associations in GPT — Meng et al., 2022. NeurIPS. Causal tracing of where facts are stored and how to edit them.
Transformer Feed-Forward Layers Are Key-Value Memories — Geva et al., 2021. EMNLP. Views MLP layers as memories of patterns.
Towards Automated Circuit Discovery for Mechanistic Interpretability — Conmy et al., 2023. NeurIPS. Automating circuit finding (ACDC).

Toy Models of Superposition — Elhage et al., 2022. Anthropic, Transformer Circuits. Why neurons are polysemantic and how superposition works in small models.
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Bricken et al., 2023. Anthropic, Transformer Circuits. Sparse autoencoders find interpretable features in a small transformer.
Sparse Autoencoders Find Highly Interpretable Features in Language Models — Cunningham et al., 2023. arXiv. Independent SAE results.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Templeton et al., 2024. Anthropic, Transformer Circuits. SAEs applied to a production-scale model.
Scaling and evaluating sparse autoencoders — Gao et al., 2024. OpenAI. Methods and metrics for large SAEs.
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 — Lieberum et al., 2024. Google DeepMind. Open SAE suite for researchers.

Circuit Tracing: Revealing Computational Graphs in Language Models and On the Biology of a Large Language Model — Anthropic, 2025. Transformer Circuits. Attribution graphs for studying multi-step computation in a large model.
Mechanistic Interpretability for AI Safety — A Review — Bereska and Gavves, 2024. arXiv. A broad survey of the field and its safety relevance.
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models — Rai et al., 2024. arXiv. A practitioner-oriented review.
Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al., 2023. arXiv. A related top-down approach (reading and steering representations).
200 Concrete Open Problems in Mechanistic Interpretability — Neel Nanda. Alignment Forum. A source of research project ideas.

Author: mej7.com, an independent educational project. Last reviewed: 6 November 2026.