Interpretability for AI: How to understand neural networks

Who should read it?

Technically curious readers, data scientists, and managers who have to sign off on models they cannot inspect

Intro

In 2017, researchers trained a neural network to classify chest X-rays and it performed well. Too well, as it turned out. When someone finally looked at which parts of the image the model was reading, it turned out the model paid attention to the Chinese characters printed in the corner of the scans, because most of the positive cases in the training data had come from hospitals in China. The model had learned to detect the hospital, not the disease.

That story is the best argument for interpretability, the field that tries to answer a question neural networks are not built to answer: "Why did you decide that?" The network cannot tell you. Nobody wrote the decision path. Learn about how researchers read one out of the machine anyway, and where the reading stops being science and starts being careful guesswork.

We defined the what and gave up the how

Classical software is a recipe. A developer writes every step, so when the output is wrong, the mistake is somewhere in the steps and you can go find it. Neural networks broke that contract. We no longer specify the solution path; we specify the goal and a training procedure, and the network finds its own path to the goal. We kept control of the what and handed over the how.

That trade is the reason the technique works at all. Nobody knows how to write, as an algorithm, the rule for "which of these 14 million pixels belong to a pedestrian". We give examples, the network finds its own rule, and the rule turns out to be better than anything we could have written. The price is that the rule lives somewhere no one planned to store explanations.

Architecture nudges the how without determining it. Convolutional networks process images in small local windows and stack those windows into layers, so global structure only emerges gradually. That shapes the kind of solution the network tends toward, but it guarantees nothing about being understandable.

The real problem is size. A modern language model turns each token, each fragment of text, into a vector with thousands of dimensions; DeepSeek's open model uses 7,168 (arxiv.org/abs/2512.02556). Asking whether the vector for "cat" sits near the vector for "dog" sounds like a reasonable question until you remember that "near" in 7,168 dimensions has no picture a human can hold in their head. Everything interpretability produces is a projection of that space into something we can look at, and every projection loses information.

An MRI for the artificial brain

Neuroscience faced the same problem decades earlier and found a workaround. Nobody can trace a single synapse in a living brain, but you can put a person in an MRI scanner, show them pictures of cats and cars, and watch which regions light up. Patterns in the activation give you a functional map: not the wiring, but a map of what responds to what.

Interpretability research borrowed the idea wholesale. You feed inputs into a network and record which units respond, then compare across inputs. If the same region responds to dogs and leopards but not to sedans, you have learned something real about what the network tracks, even though you still cannot name the feature it computes. The map is coarse, and like all coarse maps it can mislead, but it is the difference between poking the box with a stick and watching how it behaves from the outside.

Faces in toast, dogs in clouds: DeepDream

Humans have a version of the same trick built in. Pareidolia is the reason you see faces in electrical outlets and Jesus in toast: the brain amplifies faint patterns until they become recognizable. Show anyone the face of the Man in the Moon and they cannot unsee it. Psychedelics push the same amplifier harder, which is why LSD turns wallpaper into snakes.

DeepDream (research.google/blog/deepdream-a-code-example-for-visualizing-neural-networks) applies the amplifier to the network itself. Pick an output, say the "dog" class, then adjust the input image step by step to raise the network's confidence that the image contains a dog. The image drifts into whatever the network actually means by dog: eyes and snouts bloom out of clouds and foliage, because those are the features the internal layers respond to. Run it on a single layer instead of the output and you get that layer's preferred stimuli, a kind of layer portrait.

The images are famous mostly as internet weirdness, which undersells them. They are evidence. The network is showing you what it looks for, in the only language it has.

Heatmaps for opaque decisions

The X-ray problem from the intro has a general solution. Take an image and a text the model is generating, then measure how much the prediction changes when you perturb each pixel. If the current text is "A malignant mass on the" and a tumor is visible in the liver region, the probability of the next token being "liver" should depend heavily on pixels in that region. Do this for every pixel and you get a heatmap of influence.

For a radiologist working with the model, the heatmap is the difference between using a tool and trusting a stranger. It shows whether the model is reading the tumor or reading the hospital label. The 2017 story above came from exactly this kind of check, and the uncomfortable part is that the check was needed at all. The model was accurate. Without the map, accuracy alone would have shipped it.

The method is deliberately naive, and that is part of its honesty. It makes no assumptions about the network's internals; it just asks the black box the same question over and over with one thing changed at a time. What comes back is influence, not mechanism. The heatmap tells you where the model looked. It does not tell you what it saw.

Inside the thoughts of an LLM

Language models can be opened further than images, because their structure is more regular. Nearly all of them are transformers (arxiv.org/abs/1706.03762): dozens of stacked layers that alternate between mixing and processing. In the mixing step, attention lets every token pull in information from every other token. In "black cat", part of the "black" token's meaning flows into "cat". In the processing step, each reshaped token is transformed on its own, which is where factual knowledge gets applied. Mix, process, mix, process, dozens of times.

That regularity gives you 3 ways in, roughly in order of difficulty.

1. Attention itself is inspectable
   1. You can record which tokens paid attention to which, and the records answer concrete questions: "Does the model lean on a typo it memorized during training?", "Does it connect a claim back to the sentence that supports it?". The visualization looks like threads running between words, and sometimes the threads show something the model's authors did not intend.
2. Many large models are mixtures of experts
   1. The processing layers are split into subnetworks and a router sends each token to one of them. Routing decisions are logged by design, so you can simply observe that expert 5 handles most of the mathematics while expert 8 handles French. Nobody taught the experts those specialties; they emerged, and the router makes them visible.
3. Most interesting, you can go looking for concepts
   1. As an input moves through the layers it lives in that 7,168-dimensional space, and some regions of the space turn out to cluster: logic and numbers live near each other. Better still, researchers have found that the model appears to think ahead. Ask it to write "If you add 5 and 3 together, the result is" and traces of the state describing "8" are already present in the earlier tokens, before "8" is written. The model is not just predicting the next word; parts of the answer exist upstream of the answer.

None of this is clean. The structures are statistical tendencies, not rules. Nothing in training forces a network to keep French in expert 8, and no mechanism guarantees that "logic" stays in one region of the space. The network optimizes the task; any tidy internal organization is something it happened to find useful, not something it owes us. Deviations and exceptions are expected, which is why interpretability claims are usually written in the language of "tends to" and "often".

One response is disentangling. Instead of hoping the network organizes itself legibly, you retrain it with an incentive to keep concepts separate. OpenAI describes the approach in its post from Nov/13/2025 on sparse circuits (openai.com/index/understanding-neural-networks-through-sparse-circuits). No instruction like "neuron 43 in layer 8 shall represent French wines", only pressure toward separation, after which individual features become addressable and, in principle, editable. That last part matters: if you can locate a behavior, you can sometimes suppress it, which moves interpretability from observation toward repair.

What it is good for, and what it is not

Interpretability earns its keep in debugging and safety. The X-ray model passed every accuracy metric and still would have killed people under the wrong deployment. Heatmaps, attention traces, and feature attribution catch exactly this class of failure: the model optimizing a proxy, latching onto metadata, or relying on a correlation that will not hold. The same logic applies to any image classifier. Ask whether a sheep detector is looking at the sheep or at the green meadow and the fence behind it, and the answer decides whether the model still works in a different field next month.

It also increasingly matters for everyone who has to approve a model without building one. Regulators ask why a system decided what it decided. Customers ask. Courts ask. A team that can produce an influence map has a different conversation with all three than a team that can only repeat the accuracy number.

What it is not, yet, is understanding. Everything described here is diagnosis from outside: patterns observed, correlations measured, structures inferred. The fundamental problem stays the same as in the intro. We are looking for legible structure in an object too large and too high-dimensional to be legible, and what we find are estimates, always. The model itself has no hard boundary between "mathematics" and "French"; the boundary is something we draw onto the map.

An instrument, not a proof

Interpretability will not turn a neural network into a recipe. That is not the goal worth having. The goal is narrower and more useful: to see where the machine is looking, to notice when it looks in the wrong place, and occasionally to reach in and stop it. DeepDream shows what a network wants to see. Heatmaps show whether it is checking the tumor or the hospital label. Attention traces and expert routing show what a language model is actually leaning on when it answers.

And the accumulated picture is worth having even without any single finding: these systems are stochastic machines, capable and strange, which resolve problems in ways we can sometimes observe and rarely predict. A field that runs on systems nobody can fully read has a choice between pretending otherwise and building instruments. Interpretability is the instrument. It has more in common with a vague diagnostic scan than with a proof, and it is still the difference between engineering and alchemy.

The 5 most important takeaways

1. Neural networks learn the "what", not the "how". The solution path is neither specified nor guaranteed to be comprehensible.
2. Interpretability works indirectly and statistically. DeepDream, heatmaps, and attention analyses show patterns and reactions, not clear thoughts.
3. High dimensionality makes direct understanding impossible. Internal representations with thousands of dimensions can only be grasped through lossy projections.
4. Interpretability is a safety instrument. It exposes proxy targets and hidden correlations, such as models reading the hospital instead of the disease.
5. AI remains a stochastic machine. Even with interpretability, there are no hard semantic boundaries inside the model.