Mechanistic Interpretability MOC

Mechanistic interpretability tries to understand neural networks by identifying the internal algorithms, features, circuits, and causal pathways that produce behavior.

Why It Matters

Frontier labs care because:

Models are powerful and opaque.
Behavioral evals do not reveal mechanisms.
Safety failures can hide until deployment.
Understanding features may enable monitoring, steering, debugging, and assurance.

Core Ideas

Residual Stream

The residual stream is the shared workspace across transformer layers. Attention and MLPs write updates into it.

Connects to:

Circuits

A circuit is a set of components that causally produce a behavior.

Examples:

Induction heads.
Copying circuits.
Name-mover heads.
Arithmetic or factual recall pathways.

Primary source:

A Mathematical Framework for Transformer Circuits

Superposition

Models may represent more features than dimensions by placing features in superposition.

Primary source:

Toy Models of Superposition

Sparse Autoencoders

Sparse autoencoders try to decompose activations into more interpretable features.

Primary sources:

Causal Interventions

Techniques:

Activation patching.
Path patching.
Ablation.
Feature steering.
Logit lens.
Direct logit attribution.

The point is causality, not just correlation.

Starter Project

Replicate a tiny induction-head analysis:

Use a small transformer.
Create repeated-token sequences.
Identify attention heads that copy previous-token patterns.
Patch or ablate the head.
Measure change in next-token prediction.

Deliverable:

A memo explaining what behavior the head causally supports.

Papers / Resources

What Good Looks Like

You can:

Explain a transformer behavior in terms of internal activations.
Distinguish correlation from causal evidence.
Run activation patching.
Train or use a sparse autoencoder.
Write down what would disprove an interpretability hypothesis.