Mechanistic interpretability is the study of how neural networks compute their outputs by reverse-engineering internal mechanisms such as features, circuits, attention heads, activations, and weights. It aims to explain what a trained model is doing internally, not only whether its output is correct.
In Large Language Models (LLMs), mechanistic interpretability research often investigates:
- which internal features correspond to concepts or behaviors;
- how attention heads move information through the residual stream;
- how circuits implement tasks such as copying, induction, or refusal;
- whether a feature causally affects model output; and
- how model internals change under fine-tuning or steering.
Mechanistic interpretability is different from ordinary evaluation. An evaluation measures external behavior. Mechanistic interpretability tries to identify internal computations that produce that behavior.
The field is technically difficult because modern models have billions or trillions of model parameters, distributed representations, and nonlinear interactions across layers. Results that explain a small circuit may not generalize to all model behavior.
Anthropic's Transformer Circuits Thread collects research on reverse-engineering transformer internals and related interpretability methods.
The LLM Knowledge Base is a collection of bite-sized explanations for commonly used terms and abbreviations related to Large Language Models and Generative AI.
It's an educational resource that helps you stay up-to-date with the latest developments in AI research and its applications.