Home 零对冲(ZeroHedge)每日HackerNews

神经网络已经干净
Neural networks are clean

原始链接: https://already-clean.goodfire.com/

经过多年研究神经网络，作者得出结论：与以往领域的认为相比，这些网络实际上远具可解释性。曾被认为是任意的多义性——神经元对无关概念的响应——现在被认定大多是递归泛化，神经元沿着如树枝般的相关维度生长（例如检测 'iy' → 'diy' → 'do it yourself'）。主导的超位解释——由于干扰不足而将无关事物打包在一起——在现代工具和架构下成了误读。借助更好的可解释性工具和架构改进（残差连接、层归一化），现代神经网络在最后一层揭示了 99-100% 可解释的神经元。该领域最初直接研究网络，但在早期模型似乎不可解释时退回到特征提取；如今，借助合适的工具，直接研究同样有效。网络使用单一的学习原理（泛化），而非两者，随着解释图填充完整，它们变得愈发可读。较大的模型倾向于拥有更窄、更专化的神经元。作者主张不应过度依赖清洁度评分或特征字典，而是应直接研究神经元和电路，将其视为网络自身的可读构块。恰当观察时，几乎每个机制都说得通，这使得可解释性比假设的那样更易于处理。

Nick Cammarata's blog post "Neural networks are clean" argues that modern neural networks are far more interpretable than previously believed, challenging the prevailing view of polysemanticity (neurons encoding multiple unrelated concepts). Cammarata contends that with advanced tools and closer inspection, the vast majority of neurons in models up to trillions of parameters are legible, with beautiful, interpretable circuits. He claims many previously cited examples of "tangled" neurons were misunderstandings, and that the roadblocks he once expected largely don't exist. The post suggests ambitious interpretability may be solved sooner than thought, as automated tools improve. Commenters debate the implications: some see this as a potential watershed for AI alignment by making model internals more understandable, while others remain skeptical about whether this translates to solving deeper alignment problems. Discussions touch on emergence, superposition, and whether "clean" neurons truly simplify model understanding or merely shift interpretive challenges. The core claim is that neural networks' interpretability outlook is significantly more positive than the field had assumed, particularly for large-scale models.
相关文章
原文

The Short Version

I think I have good news about the nature of neural networks.

After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning. This means we can study networks directly, instead of first untangling them. Neural networks are already clean.

There are at least two ways to approach mechanistic interpretability. The first is rapid normal science on the network itself: look at a network’s neurons and circuits, study how they relate, watch data flow through them, and run experiments at agentic scale to test explanations. The second starts from a list of hypothesized features and tries to extract them all, designing increasingly sophisticated reconstruction objectives and usually ignoring the original.

The field began with the first approach and fell back to the second because neural networks seemed to be tangled knots. We had canonical examples of polysemantic neurons, the broad name for a neuron that appears to have multiple unrelated jobs or that looks impossible to understand. The most famous one, in an image model, responded to both cat faces and the fronts of cars. And the last layer of the early field’s model organism, InceptionV1, seemed so full of arbitrary mixtures it wasn’t clear how to study it: almost none of the neurons were easily interpretable.

The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing. You have to unfold the knot before you can study anything. In practice that means training a second model – a decomposition such as a sparse autoencoder – to approximate the network as it would look unfolded and studying that instead. If the network really is knotted, there may be no alternative.

One main result is that superposition was the wrong explanation for InceptionV1. Architectural problems made later layers overflow. Weak interpretability tools amplified these problems, making the late layers seem incoherent. Since then, architectural changes designed to help deep layers learn better have had the unintended effect of making them more interpretable. When we first studied InceptionV1, we found almost no interpretable neurons in its last layer. But with these ordinary improvements, and no change to model size or training data, 99-100% of neurons in that last layer are easily interpretable and the model is broadly legible the whole way through. It’s the first non-trivial neural network I feel I mostly understand; you can browse it here. I trained versions of it across a 300× range in size, and it’s broadly legible at every scale. With less room, the network doesn’t fold unrelated things together; its neurons just cover coarser-grained concepts, like one car neuron covering many kinds of car. With more room, that neuron differentiates into many specific cars. So the problem wasn’t the presence of a second learning principle, it was just an ordinary learning failure of the first one, and it’s easy to fix.

The canonical vision examples of polysemanticity and superposition were errors too. In 2019, Chris Olah and I found the circuit that was originally introduced as the superposition phenomenon in Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓. It showed a car feature appearing to spread into a set of unrelated dog neurons. It was the only one we showed, and the only one I ever saw. We read it as the network placing the car in a high-dimensional hiding place, for downstream neurons to read back out. But nothing reads that combination. It wasn’t hiding anything; it was ordinary generalization and composition. To my knowledge, no one has published a second circuit with the right shape.

Modern language models are also far more interpretable than generally believed. Anthropic’s Towards MonosemanticityTowards Monosemanticity: Decomposing Language Models With Dictionary Learning
Bricken, T., Templeton, A., Batson, J., et al., 2023. Transformer Circuits Thread. References ↓ (2023), studying a small one-layer model, gave the median neuron a 0 on its rubric out of a possible 14, meaning the rater couldn’t form any hypothesis. The automated interpretability projectLanguage Models Can Explain Neurons in Language Models
Bills, S., Cammarata, N., Mossing, D., et al., 2023. OpenAI. References ↓ I started at OpenAI (also 2023) found about 1.7% of neurons interpretable by our most generous measure.

With better tools and modern models, the situation looks quite different. In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understoodA note on this number: the share of neurons with one job varies depending on how you set the threshold for counting weak activations, depth, density of sub-circuits, and what you view as “one thing”.

How far to go into weak activations is a big factor here. There’s a section describing weak activations below. It tends to increase how many “related things” there are, more than whether the neuron is a judgement of something legible. For instance, the Trump neuron’s weak activations bring in MAGA hats and people related to him. The Titanic neuron’s weak activations bring in stories of more general voyages, or steam engines and other technologies of the era.

There’s also the question of whether to care about activation percentiles, or fraction of the strongest activations. In general I think what the neuron learns should be trusted, its natural sparsity and activation ranges and so on. Which I used to mostly not believe! I’m somewhat against trying to standardize things across all neurons. Percentiles are more common today, I think because they’re easier to think about.

It goes the other way too. For instance, when you look at the Titanic neuron’s “unrelated jobs,” it’s often detecting that “boat” comes next. The tool missed this because it only looked backward when judging context. When we look at these “unrelated jobs” in depth, it’s almost always this kind of thing. It’s easy to build an imperfect tool.

Overall, I don’t think coming to a single score is actually that important. But when it was taken out, early reviewers felt unanchored without even a rough number. In the sets I’ve looked at more closely, considering strong activations, it’s closer to 97% non-arbitrary mixtures, but these are relatively small samples.

Overall distilling everything into one “cleanliness” score should be taken as a broad pointer to the regime a neural network is in, rather than something precise to either optimize directly or hold too tightly. A lot of this essay is for being less dimensionality reduced, treating each neuron and circuit differently, according to their motifs, depth, role etc. Despite this, a rough number can be useful.. Dense and Mixture of Experts models seem roughly similar. We found similar results in models up to trillions of parameters, like Kimi K3, which the companion paper will cover in more detail. These percentages compress a high-dimensional picture into a few numbers, and the precise number depends on where you set thresholds for what counts as “one thing” – like whether you lump together closely related idioms in two languages. I don’t think much in interpretability hangs on that question. What matters a lot more is why things share a neuron in the first place.

Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. This idea first appeared in a forgotten subsection of a paper I worked on called Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓ (2021). At the time, I hadn’t watched neurons grow over training. Now it’s easier, and it holds up everywhere I’ve seen. Even the original InceptionV1 wasn’t knotted. When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.

So there were two explanations for polysemanticity, and I helped propose both. In some ways, they’re opposites: generalization groups similar things together and superposition groups unrelated things together. Superposition is the one the field bet on and, until recently, so did I. But I now think recursive generalization was basically right. That leaves one learning principle after all. It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more. In that sense, almost the entire network is interpretable if we look closely enough, especially if we look at how it grew over training.

Why was the gap so large? I think several ordinary problems compounded. The explainer models of 2023 couldn’t run the loop of conjecture and test; the tools and metrics were weak, rewarding explanations that could simulate activations rather than ones that were right, which often come apart; no one knew what language neuron motifs to expect, and people broadly looked for the wrong kinds of things; and most studies used small or simplified models. In a sense, those models really had run out of room, which made superposition the natural explanation. But a network short on room doesn’t switch learning principles; its neurons just get broader, as in the small vision models. Broad neurons are usually easy to break down once you know what to look for. Without that knowledge, and with primitive tools, they look like arbitrary mixtures.

Interpretability has a surprising amount of momentum: like a jigsaw puzzle or crossword, any understanding you gain starts constraining the rest of the system. At first, you’re looking at a single neuron. If you can’t understand it, little else helps: circuits and attribution graphs only explain a neuron in terms of other neurons you don’t understand yet. This can make getting started hard. Once the map starts to fill in, though, it goes surprisingly well. If you don’t understand exactly what a neuron does, you can study what it connects to, the ecosystem around it, and – in a mixture-of-experts model – what its expert specializes in. As with a puzzle coming together, every piece comes with many clues, and they’re well structured and easy to run experiments against. That’s why nearly empty and mostly filled in aren’t surprising places to be; they’re the natural anchors.

Take a neuron that tracks ‘a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative’. Neurons that track a feeling or social cue like this are some of the most common we’ve found. Imagine finding this one with no knowledge of language neuron motifs and no circuits to lean on, because few neighboring neurons are explained yet. Most humans and agents expect neurons to detect nouns by firing on them, which they mostly don’t. Look at where this one fires most strongly, and you see a pile of seemingly unrelated commas.

Now imagine the map is mostly filled in. You know many neurons detect subtle feelings or social contexts, and that they often fire not on the words that carry the feeling but on punctuation, like commas, where the model summarizes what a sentence meant, or on the word just before the feeling matters for predicting what comes next. So you read the surrounding sentences instead, and check its steering effects (what the model writes when you turn the neuron up), which line up closely. You look upstream at how it’s built and downstream at what it does, and both line up. You check its negative pole, what pushes it strongly negative, and find a subtle variant of the same cue, or the moment the social context ends. You can run specific tests, like checking which store brands it responds to. Any of these could break the explanation, yet they keep pointing to the same place. And understanding this neuron sharpens the broader motif: after thousands of subtle-feeling and social-context detectors, you start to know what to expect from the next one, which is part of why the map fills in faster as it goes.

It works on the tools too. When you can see 2% of a network, you can’t tell where your tools are broken; when you can see most of it, the unexplained remainder points straight at them. That’s how some coworkers and I found a double-negative regime in SwiGLU neurons that, as far as we know, no one had noticed, and that a decomposition mixing many neurons together would likely have hidden.

The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps: as the map fills in, every explanation is tightly constrained by everything around it, and bigger models learn narrower neurons that are easier to recognize in the first place.

In this sense, understanding the whole network may be easier than understanding just the parts we care about most. First, it’s hard to know how to think like a neural network until you’ve studied it closely. For instance, “deception” is a useful initial frame, but the building blocks that contribute to deceptive behavior by models probably have alignment-relevant microstructure that will be hard to know without going in and looking. Second, targeting a small set of mechanisms is like working on one row of an otherwise empty crossword. It’s doable, and might work, but without the crossings it’s hard to know whether you’re right. It may be easier to solve the whole thing – or at least the broader parts of the network these mechanisms connect to.

When I ran this process on modern convolutional networks, it basically worked. Agents can map out neurons and circuits, build explorable microscopes and explore them, test explanations thoroughly, and track hundreds of thousands of attribution graphs. There was never a point where rapid normal science broke down. I now feel I basically understand these models, at least enough to know where I’d start asking any question I had; eventually, the crossword just got filled in. So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved due to sheer size, but I’m optimistic about scaling the process.

All of this has brought me back to an old hope: understanding a neural network directly, by studying its mechanisms and the data flowing between them, without first trying to fix or untangle it. Put together, the argument has four steps.

  1. Networks use one learning principle, not two (if not everywhere, then nearly enough that it doesn’t matter for study), so there’s no blocker to beginning ordinary science on them.
  2. With modern models and tools, that science goes much better than most people expect.
  3. Understanding compounds, so it gets easier as the map fills in. At minimum, then, a network’s own mechanisms should be part of the toolkit.
  4. Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed.

That’s been the pattern for deep learning in general: the simple thing at absurd scale beats the clever one. Rich Sutton called it the bitter lessonThe Bitter Lesson
Sutton, R., 2019. Incomplete Ideas. References ↓. Maybe there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition.

A technical companion paper will offer a more detailed account of language model neuron experiments, such as what we’ve found about the different regimes of modern non-linearities and what kinds of circuit motifs show up at different depths. This essay glances at many places and conjectures about where it all might be pointing.

Cars and Animals

The most famous polysemantic neuron is 4e:55 of InceptionV1, which looked like an arbitrary mixture of cats and cars. It’s been brought up as evidence that neurons might not be the right unit of study in several of the field’s most widely read papers, from Feature VisualizationFeature Visualization
Olah, C., Mordvintsev, A. and Schubert, L., 2017. Distill. References ↓ in 2017 to Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓. In Zoom In, which I co-authored, we said outright that it “isn’t responding to some commonality of cars and cat faces”.In Multimodal Neurons we came very close. We asked whether the neuron might be responding to some shared feature, “perhaps cat eyes and car lights look similar,” then decided from the feature visualization’s whiskers and chrome that it was “genuinely OR(cat, car).” We were right to look for a shared shape, but I think we underestimated how far generalization reaches. The shared feature was the V. I think the whiskers misled us because generalization happens inside high-level patterns. The V neuron really does take in cat circuitry, and feature visualization’s job is to raise the activation, so once it finds the cat route it adds everything that route brings. The whiskers come along for the ride, even if the neuron doesn’t “want” them. Downstream, the car readers do get a little cat signal through the V neuron, but it barely reaches the output: one cat detector firing does little on its own, since many work together, so the network never needs a negative weight to cancel it. To see the shared pattern, you have to visualize what the neuron’s inputs have in common, not the neuron on its own.

After months of seeing clean neurons, I wondered if we’d misinterpreted some early prominent examples of polysemanticity and superposition. The canonical polysemantic neuron seemed like the obvious place to check, so I opened up the cat and car image grid I’d seen hundreds of times before and did something I’d never done: I spent a few seconds wondering what one thing this might be. Almost instantly it was obvious: it’s not a cat and car neuron, it’s a V-shape neuron.

I looked at its circuit connections: V-shaped flames, V-shaped water jets from a fireboat, V-shaped fox faces. I looked at its negative pole: an O, round where the V is pointed. Why would an arbitrary mixture of cats and cars need to connect so strongly to a flame?

Above, its strongest inputs and readers (line width is strength) and generic renders. Below, what it sees in each photo: pointed things drive it and round ones push it negative, whatever the label.

Then I went to the canonical superposition circuit I first found in 2019I found it while building my first Circuits interface, which only had feature visualization and weights connecting their neurons. Then I took them to Chris Olah and asked why the car would be going into these dogs like this. We came to the conclusion it must be doing a kind of folding, and called it superposition. It was an intuitive explanation for what we saw, so it didn’t prompt diving more into the tools, which would have revealed feature visualization reward hacking and various other things. It being placed into a high-dimensional space felt like a reasonable answer.. The V neuron and this circuit were Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓’s two main examples of why its picture of neurons, that nearly every one turns out to be understandable if you look seriously, might be “overly rosy”.

The circuit appeared to show a car going into a bunch of unrelated dogs, as if the dogs were a kind of high-dimensional storage place where the car could be recovered later as needed.

But that’s not what the car was doing. Cars and dogs were being brought together because of shared pose. InceptionV1 has more than a hundred of these generalized pose neurons, each one its own clean detector of a pose shared across cars, dogs and other objects, and to my knowledge no one had noticed them. Later layers keep building on them, all the way to the end of the model. The car was not being stored in an obscure place, and nothing directly reads out the combination. It’s the opposite: it’s continuing to differentiate into car specialists and finer poses.

Part of the reason was not knowing about generalized pose neurons, and part of the reason is that the technique we use to render icons of images — feature visualization — ends up being reward hacked in deeper layers. The intent is to find an image that maximizes the neuron, but the way that’s done is through gradient descent, and it takes the easiest path, which in InceptionV1 often means dogs. There are lots of dogs in InceptionV1’s training data and a lot of dog neurons, and the early optimization falls into them and never recovers. If you even lightly bias it toward the natural paths the top images take, most of them are cars, and some are both dogs and cars, because of shared pose.

A History of Microscope Errors

The story of interpretability, as I see it, has largely been one of finding out that what was under the microscope was better than we thought. The microscope just wasn’t tuned right, and what needed tuning was often pretty subtle.

Modern neural network interpretability arguably started with feature visualization, which renders a synthetic image of what a neuron responds to most: the famous psychedelic pictures. For a while they didn’t look very good. You could reasonably conclude the neurons had learned something, just nothing very clean. It turned out the method was missing transformation robustness. Once researchers added it, the same neurons came out crisp.

But that only gets you so far into the model. Near the end, feature visualization starts to break down again, for two reasons: the reward hacking we saw with the car circuit, and regularizations that weren’t quite right. Fixing it didn’t require anything fancy. A few subtle tweaks to reasonable settings, and now we can see much deeper into the network. Same neurons, better microscope.

The same two InceptionV1 neurons, beside their strongest photos. Small regularization tweaks bring out first the middle neuron, then the late one.

But it wasn’t all the microscope. The model needed to be tuned too. InceptionV1, the early model organism of the field, was totally uninterpretable in the late layers. One estimate said “To the best of my knowledge, there are essentially no interpretable neurons on InceptionV1’s final layer,” which is close to my own recollection, I’d have maybe said 5%. This is what they looked like.

Feature visualizationDataset examples
A late-layer InceptionV1 neuron, shown through optimized views and natural images.

The leading explanation as far as I knew was that there were too many high-layer features needed to be represented for the number of neurons and parameters there were, so it started packing ones that don’t belong together arbitrarily, since they don’t collide. The early layers were monosemantic because not as many early layer features needed to be represented.

But this wasn’t the right explanation. Here’s a more modern convolutional network with the same number of parameters and dataset, but with residual connections, LayerNorm, and a couple other improvements. The last layer is beautiful.

Ladybugs · Neuron 127

Each neuron’s 20 strongest responses out of 50,000 images, in order. Labels mark images from other classes.

So two things were required. First, the neural network needed to learn well, using an architecture designed by people fully focused on that goal and with no interest in interpretability. The residual stream is one example: it was introduced to help learn deeper neural networks, and it also seems to help the deeper layers be conceptually clean, which may be related. Second, we needed tools strong enough to see them clearly.

The last layer of InceptionV1 was one of the main pieces of evidence that neural networks were almost fundamentally uninterpretable, since the way they learn overlaps unrelated mixtures. Replacing the neurons seemed like the only option. I put it around 5% with tools of that era, others at ~0%. With better interpretability tools (fixed feature visualization, methods for visualizing attribution etc) it’s more like 50% on the same network. This is all with the 2014 model whose checkpoints and training details were lost to time. Retraining the same architecture with more reasonable modern regularization it’s more like 70%. In the ConvNeXt of the same size, it’s about 99%. They’re explorable here.

Recursive Generalization

So we’ve shown the last layer was fixable. But what was it doing before it was fixed? Even before the architectural improvements, it wasn’t putting things together arbitrarily. When we retrained the model and watched a neuron form, it started on yellow flowers, in photos that sometimes included a dog. Then it grew along each facet of those photos: toward other kinds of nature, toward green objects like balls, and toward dogs without any flowers. This is true of every neuron I looked at. The improved architecture shows the same pattern, but much less of it.

The grey band behind each row is its share of the neuron’s strongest images.

Two neurons over training, from epoch 0.7 to 12, one row per group of images. InceptionV1 neuron 338 starts on rapeseed fields and valleys, with one dog sitting in a rapeseed field. By epoch 1.5 it also responds to tennis balls, green apples and jackfruit, and dogs in grass and yellow flowers; by epoch 3 to shaggy dogs, often with a tennis ball, and broccoli; by epoch 12 it is mostly tennis balls, terriers and broccoli, and the fields have thinned to a sliver. ConvNeXt neuron 399 starts on the same fields and also briefly responds to golf and tennis balls and fruit, then drops them; by epoch 12 it is rapeseed fields, plus one dog lying in yellow flowers. epoch 0.7 1.5 3 6 12 InceptionV1trained from scratch · last layer, neuron 338 rapeseed rapeseed rapeseed rapeseed valley rapeseed rapeseed rapeseed tennis ball tennis ball tennis ball tennis ball tennis ball Granny Smith acorn jackfruit acorn acorn EntleBucher Appenzeller Newfoundland Norwich terrier curly-coated retriever komondor Bouvier des Flandres Dandie Dinmont cairn Bouvier des Flandres broccoli broccoli broccoli cauliflower fields and valleys tennis and golf balls fruit and acorns dogs broccoli and cauliflower ConvNeXtsame data and recipe, with LayerNorm and residuals · last layer, neuron 399 rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed golf ball tennis ball tennis ball soccer ball tennis ball Granny Smith orange jackfruit Kerry blue terrier Appenzeller Appenzeller fields and valleys tennis and golf balls fruit and acorns dogs broccoli and cauliflower Two neurons over training, from epoch 0.7 to 12, one row per group of images. InceptionV1 neuron 338 starts on rapeseed fields and valleys, with one dog sitting in a rapeseed field. By epoch 1.5 it also responds to tennis balls, green apples and jackfruit, and dogs in grass and yellow flowers; by epoch 3 to shaggy dogs, often with a tennis ball, and broccoli; by epoch 12 it is mostly tennis balls, terriers and broccoli, and the fields have thinned to a sliver. ConvNeXt neuron 399 starts on the same fields and also briefly responds to golf and tennis balls and fruit, then drops them; by epoch 12 it is rapeseed fields, plus one dog lying in yellow flowers. epoch 0.7 1.5 3 6 12 InceptionV1 trained from scratch · last layer, neuron 338 rapeseed rapeseed rapeseed rapeseed valley rapeseed rapeseed rapeseed tennis ball tennis ball tennis ball tennis ball tennis ball Granny Smith acorn jackfruit acorn acorn EntleBucher Appenzeller Newfoundland Norwich terrier curly-coated retriever komondor Bouvier des Flandres Dandie Dinmont cairn Bouvier des Flandres broccoli broccoli broccoli cauliflower fields and valleys tennis and golf balls fruit and acorns dogs broccoli and cauliflower ConvNeXt same data and recipe, with LayerNorm and residuals · last layer, neuron 399 rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed rapeseed golf ball tennis ball tennis ball soccer ball tennis ball Granny Smith orange jackfruit Kerry blue terrier Appenzeller Appenzeller fields and valleys tennis and golf balls fruit and acorns dogs broccoli and cauliflower Two neurons over training, from epoch 0.7 to 12, one row per group of images. InceptionV1 neuron 338 starts on rapeseed fields and valleys, with one dog sitting in a rapeseed field. By epoch 1.5 it also responds to tennis balls, green apples and jackfruit, and dogs in grass and yellow flowers; by epoch 3 to shaggy dogs, often with a tennis ball, and broccoli; by epoch 12 it is mostly tennis balls, terriers and broccoli, and the fields have thinned to a sliver. ConvNeXt neuron 399 starts on the same fields and also briefly responds to golf and tennis balls and fruit, then drops them; by epoch 12 it is rapeseed fields, plus one dog lying in yellow flowers. epoch 0.7 1.5 3 12 InceptionV1 trained from scratch last layer, neuron 338 rapeseed rapeseed valley rapeseed tennis ball tennis ball tennis ball tennis ball Granny Smith acorn acorn EntleBucher Appenzeller Norwich terrier Dandie Dinmont cairn broccoli broccoli cauliflower fields andvalleys balls fruit andacorns dogs broccoli ConvNeXt same data and recipe, with LayerNorm and residuals · last layer, neuron 399 rapeseed rapeseed rapeseed rapeseed rapeseed golf ball tennis ball soccer ball Granny Smith orange jackfruit Kerry blue terrier Appenzeller fields andvalleys balls fruit andacorns dogs broccoli
A neuron in InceptionV1 and one in ConvNeXt, trained on the same data, that both start out on rapeseed fields. Both reach for balls and fruit early on; ConvNeXt lets them go and stays on the fields, while InceptionV1 keeps spreading, into dogs and then broccoli, until the fields are nearly gone.

We call this recursive generalization: a neuron generalizing along more than one axis, often first along one axis, then sprouting outwards in ways that are easy to follow locally, but stop it from being “monosemantic” in the end.

For instance, a language-model neuron responds to “iy,” including in “Miyazaki,” then to “diy,” and then to “do it yourself,” which no longer contains “iy” at all.

There are examples all over the place in both image and language models. Like a geology neuron that generalizes into Cesium, a geology JavaScript library.

Usually it’s very obvious what happened, to an extent where it’s hard to know whether to just call it normal generalization or not. Like an enemy neuron that fires for a Russian poem about enemies. Sometimes it’s less clear but still plausible. A Srinivasen neuron also lightly fires for the UniProt protein database. The project has several Srinivasen’s associated with it, but it’s hard to know for sure.

Top 8 firings

  1. String is a scalar string or 1-element string array, then STRSPLIT returns an array containing72
  2. determine the average of the values 0 to 9:↵SPAWN, ’test_pipe’, UNIT62
  3. Set this keyword to make the "Cancel" button the default selection for the dialog. The default selection61
  4. Stage,↵ ModelExperimentalSceneMeshPrimitive,↵} from "../../../Source/Cesium.js";↵↵describe("Scene59
  5. of KREEP-rich rock at the base of the crust left by the original crystallization of the54
  6. Highlands are older than the Northern Lowlands, based on the larger number of impact craters seen covering54
  7. ers during their final days. The dramatic footage was taken on December 14, 2054
  8. in the night sky, it's really easy to see the difference between the dark lowland plains called54

Grouped by route share of the top 100

  1. Moon and Mars crust44%comes in throughL31Mars, planetspushesHighlandsneardark side is also home to one of the largest and oldest impact features in the solar system, the South of KREEP-rich rock at the base of the crust left by the original crystallization of the
  2. Moon missions42%comes in throughL28the Moonpushesmoonfarcrater crust. But with fuel running low, NASA decided to fly them into a mountain near the Moon's the moon to distort images, the LROC (Lunar Reconnaissance Orbiter Camera) is now
  3. old impacts, Mars6%comes in throughL30life on other planetspushesplainsbasinMars exist.↵“3.4 billion years ago a large meteorite triggered the first wave of the tsunami)↵In other words, the internal pressure buildup due to the massive Hellas impact quite pos
  4. Cesium and IDL code8%comes in throughL6names like Cesiumpushes.jsobjectssoftware and precipitation rate values from which CesiumJS can then generate↵pointPrimitive objects. Because some of number of colors, once initialized, cannot be changed without first exiting IDL.↵Colormaps define

Moon and Mars crust

highlands, near and far side, the South Pole–Aitken basin

comes in throughL31Mars, planetsL28the Moon

wakes up↑ L34China’s Moon program↑ L37deep time

pushes towardHighlandsneardarkimpact

in the night sky, it's really easy to see the difference between the dark lowland plains called

Moon missions

spacecraft orbiting, imaging, and crashing into the Moon

comes in throughL28the MoonL7the Moon, early

wakes up↑ L36spacecraft, missions↑ L37impact on a surface

pushes towardmoonfarcratersatellites

but they’re running out of fuel so NASA will command them to hit the surface.↵Because it will

old impacts, Mars

Hellas, Valles Marineris, early Earth; weaker firings

comes in throughL30life on other planetsL32big volcanoes

pushes towardplainsbasinMars

pictures in one week, focusing on a crater near the western terminus of Valles Marineris

Cesium and IDL code

the CesiumJS globe library and IDL/ENVI image-processing docs

comes in throughL6names like CesiumL32numbers in software docs

wakes up↑ L37JavaScript libraries↓ L37space news

pushes toward.jsobjectssoftwarevisual

either a space or a comma is detected. In this case, IDL performs a simple string search

A planetary-geology neuron’s strongest firings mix lunar crust with IDL and Cesium code. Grouped by the circuits that feed them, they separate into three facets of one core, planetary surfaces, and one offshoot, the code, which arrives by its own route.

We’ll talk about these recursively generalized offshoots and how much they matter in the language model section. Here we just want to talk about what the motif is.

We first hinted at this in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, calling them “conjoined features.” I’d forgotten until I went back through the field’s most famous models, months into studying recursive generalization. As far as I can remember, no one, including me, suggested it might be the general explanation, for instance for what was wrong with InceptionV1’s last layer. Without training checkpoints we couldn’t watch it form, but the explanation was basically right. The appendix explains why I think the new name is more accurate.

Superposition

Recursive generalization isn’t a particularly bold explanation, it’s arguably just what we should expect out of ordinary generalization. But there’s also a much bolder explanation that has been proposed for polysemanticity, a new learning principle.

The main learning principle we see everywhere is generalization. With generalization, things are put together because of a shared pattern that it’s useful to have in one place, so it can be built on. With superposition, unrelated things are placed together because they don’t interfere. These have opposite predictions: generalization should be local, superposition should be non-local, since it would want very different things to be near each other.

Generalization would want a floppy ear dog neuron, which can be thought of as lots of slightly different floppy ears at different angles and kinds together. Superposition would want a floppy ear dog + Latin poetry neuron.

Arbitrary packing

Unrelated things share a neuron

Generalization

Similar things share a neuron

8 things per neuron
Conceptual illustration. The same 32 things appear in both panels; similar colors and shapes are similar things. Fewer neurons make generalizing neurons coarser, never fractured. Hover or tap a neuron to find its things in both panels.

The definitions get tangled here. Along with superposition, Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓ introduced the idea of features as the fundamental unit of neural networks. Depending on where you look, superposition then means either “more features than dimensions,” which even a fully monosemantic network would satisfy, because its neurons compose, or a learning principle like the one above, applied sometimes to what neurons learn and sometimes to where features go.For instance, InceptionV1’s layer 4b has basically monosemantic neurons for car parts like windows, car bodies, and wheels, which the next layer assembles into a car neuron, 4c:447. In the feature view, it’s reasonable to say the “car feature” isn’t just in 4c:447, but also in the combination of 4b neurons that builds it. So there are two car features in adjacent layers, sharing almost exactly the same mechanics. In the “more features than dimensions” view, the first is in superposition and the second may not be. In the sense of the informal note Distributed Representations: Composition & Superposition, the first is distributed, but by composition, and the second isn’t distributed at all. This is part of why I find hypothesized features hard to think with, and why I usually prefer specific mechanisms, like neurons, and specific pieces of data.

So “superposition” is itself in superposition between several common uses. We focus on the learning principle applied to the mechanisms like neurons, because whether it forms holds such a disproportionate share of the good or bad news for interpretability. It’s also fairly easy to test: recursive generalization and superposition predict nearly opposite things, and you can watch a neuron grow and see which one happens.

When I say superposition in this essay, I’m referring to this version, and we’ll address the question of “more features than dimensions”, which I basically don’t think is bad news, later.

Superposition of the Gaps

Polysemanticity is often more a claim about the observer than the neuron. It says there’s no good explanation yet. There’s a few reasons that could be:

  1. Bad tools: our lens or microscope isn’t good enough
  2. Bad explanations: created from those tools
  3. Bad learning: the network learned poorly due to bad architecture, training data, or something else
  4. Recursive generalization: while generalizing, the neuron branched off in a few related directions
  5. Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.

For years I was a staunch superposition believer. I started studying language model neurons not out of any interest in neurons or superposition, but because I was interested in whether agents had research taste! I wanted to see if they’d ask good questions as they tested explanations, and started with neurons only as a baseline. I expected them to run into intractable feature fragments in superposition. That would produce a new baseline for how clean neurons really are, then the agents could continue with some cleaner learned basis. But it never came, because the neurons kept coming, resolving to a simple explanation that made surprising predictions that held.

In a frontier language model, I found a neuron that mostly fired on “pr”, but also on a bunch of Base64 URL strings as well as some Bengali. I thought this must finally be an arbitrary mixture. But decode the URLs: “pr” words. Translate the Bengali: “pr”.

English

Please approve the project proposal.

Bengali

প্রধানprodhanhead শিক্ষকshikkhokteacher প্রথমprothomfirst প্রশ্নটিproshnotiquestion করলেন।korlen.asked

Illustration: decoded or transliterated, the noise turns out to be “pr”.

In a dense Qwen model, a coworker came across a neuron that standard top-k automated interpretability lists as a “polysemantic comma neuron”. But looking more closely, it turned out to track Colorado: phone numbers, slang, and connections to other Colorado neurons. Steering it made the model want to go hiking. The commas were where it stored a running summary of the Colorado context so far. In general attention seems to collect on the punctuation, which we call the summary motif. Once you know it, a lot of similar neurons get easier to understand.

New job, new city, new lease, a Denver address, 303 area code, green chile, Broncos Sundays, Red Rocks, bluebird powder days, Sanitas trailhead, fourteeners. The first three commas do not fire; once Denver appears, each comma fires, reaching its strongest after green chile.

My plan for this weekend is to go to the library and study for the test. I also want to visit my grandmother and help her with some gardening.

My plan for this weekend is to go to the mountains and do some hiking. I have a few different trails that I can choose from…

The commas stay dark until Colorado shows up, then brighten as it builds; steering the neuron plans a hike.

Other times, we just didn’t have enough data. At first we didn’t include long context, two paragraphs felt like enough. But this accidentally made all the long-context neurons look polysemantic.

What the agent saw

The last paragraph alone suggests a neuron for coffee and universities; the reason is further up.

In another case a neuron looked like an arbitrary mess, with nothing firing strongly. It turns out it looked for specific signs within pages of health SEO spam, and once we had a sufficiently broad dataset, it was understandable.

Across hundreds of cases like these, nothing shook my belief in superposition. It just kept moving. If the strong activations came back clean, superposition would move to the weak ones, or to combinations of neurons, or to the neurons I hadn’t looked at yet. Or maybe across layers themselves.

But eventually I wondered what was going on. Unless this was some dramatic extended case of selection bias, something wasn’t matching the early model organisms. So I went back, the original evidence fell, and that led to this essay.

I now think I was making superposition the null hypothesis when it should have been the last hypothesis, since it’s the most radical and the least testable. It just looks like things that don’t make sense. But that’s also what bad tools look like (feature visualization reward hacking), or recursive generalization you haven’t followed yet (“iy” neuron), or a bad architecture (5b of InceptionV1), or generalization you haven’t understood yet (cat and car).

I think part of what was going on was that superposition just felt right. Anthropic’s Softmax Linear UnitsSoftmax Linear Units
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ paper said as much in 2022: “In the mechanistic interpretability community, superposition is often treated as the default hypothesis simply because it seems intuitively more compelling than other explanations, but there is little evidence.”In the next sentence, the paper says its own results are moderate evidence for superposition. That evidence is circumstantial, in the paper’s own word: SoLU suppresses activations spread across many neurons, an extra LayerNorm restores performance, and after the LayerNorm a neuron’s weak activations looked less related to its main job, which the paper reads as features being smuggled through small activations. We read it as evidence about distributed representation in an architecture built to suppress it, rather than about neurons packing unrelated things together, which is the sense we care about. Telling the two apart in weak activations is hard even with today’s tools, since meaning shifts in every direction at once as activations weaken, and it was harder still in 2022, for all of us. The paper also notes that its discussion of superposition has been superseded by Toy Models of Superposition, which we address in the appendix. We bring it up not as a criticism of either paper, but because it’s the only place we know of where someone said plainly how little real-world evidence there was, and we think people still often assume there’s more than there is. To my knowledge, no one has found a superposition circuit in a real network since my original one, which was an error. I don’t see anything like it in real models, despite expecting to, and I think the bar for claiming it should be high, since it’s such a radical explanation and would be big news, terrible news, for how we should study neural networks.

Of course, this risks the opposite error, an explanation of the gaps: assuming every neuron must have a sensible story. Two things make it less worrying. First, each of the other causes can be tested. A better tool either resolves the neuron or it doesn’t; retraining with a better architecture shows whether learning was the problem; and recursive generalization can be watched as it grows, in small steps. Superposition’s only symptom is that something doesn’t make sense. Second, real explanations tend to be narrow and easily testable. Once you suspect a neuron is a Colorado neuron, steer it and see what it says. If a neuron fires for some narrow TV trope, the set of things that could count as that explanation is tiny, while “something that doesn’t make sense” covers everything outside it.

Also, we don’t have to force an explanation we don’t have. If a neuron might do a couple unrelated things and you’re not sure, it’s fine to say so. I think recursive generalization should be high on the list of things to suspect, and it’s easy to check if you have training checkpoints. I now think superposition should be far down the list.

Do Weak Activations Matter?

When people ask whether weak activations matter, I think there are several questions mixed together, and they have different answers.

Are they generalization or superposition?

Unsurprisingly, I think it’s generalization. Imagine you’re a packed, unrelated feature trying not to collide with what’s in a main neuron. Weak activations are vastly denser, so you’d be colliding with orders of magnitude more things, for little upside.

strong half

0half of maxmax

100 million tokens through one Qwen3-32B neuron. Almost all sit near zero; about 160 reach the strong half.

But maybe you hide in the weak activations of many neurons at once. What we see is that you can ablate almost all the weak activations of the network for almost no cost. The places where removing them costs the most are some of the easiest to understand. In language models, they include the last layer, which is largely used to suppress single tokens, for what we call a “say or just think” motif. In vision models, it’s the early layers, which function like dials: the difference between slightly bright and medium bright matters, but for a later detector, like king penguin, low or very low evidence doesn’t. So critical common things can’t be in the weak activations, since we can remove them.

We can also look at what we lose when we cut weak activations. Across several kinds of studies, it’s usually fine distinctions between things the neural network is already confused about. And the faint activations that tip them come from the semantically related neurons you’d expect.

Removing the weakest 75%

accuracy 72% to 71% · 1 in 10 answers change

Removing the strongest 7%

accuracy 72% to 4% · 96 in 100 answers change

The same total activation as the weakest 75%

Medium ConvNeXt, all 18 blocks. Each bar splits the model’s top two answers before anything is removed.

This leaves rare data and strong but not extremely strong activations.

Rare things, as we saw earlier, have grown on related circuits in every experiment we’ve run, since they aren’t strong enough to form niches of their own. The ablation above points the same way: if rare cases lived in the weak activations, cutting them should have broken some answers outright. Instead, every change was small, a near-tie tipping one way or the other.

And the strong-but-not-extreme activations we can just look at. In every case I know of, they radiate outward semantically: for CLIP’s Trump neuronMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, photos of Trump, then art and people related to him, then Fox News studios. The further down you go, the easier it is to be fooled by something that seems unrelated but isn’t. At minimum, look at the circuits a datapoint flowed through; what fired weakly here may have fired strongly one layer back.

Here's every strong firing of five neurons in a language model. The center is each neuron's maximum, as it radiates out it becomes more distantly related. Gray is where it's hard to place, usually these are still connected but require more work to understand, since it's more distant. You can click any to open them up and see which circuits each activation came from.

Five neurons in Qwen3-32B

Every strong firing of five neurons. The center is each neuron's maximum, rings are fractions of it, and color is what a firing is about. Gray is the floor. Click one to explore it.
").join("");} else keyEl.innerHTML=N.cats.filter(c=>c.n).map(c=>{const id=N.cats.indexOf(c);return '

'+esc(c.name)+" · "+fmt(c.n)+""+esc(c.desc)+"

"}).join(""); keyEl.querySelectorAll("div[data-c]").forEach(x=>{x.onmouseenter=()=>{ISO=+x.dataset.c;draw()};x.onmouseleave=()=>{ISO=-1;draw()}}); } function drawBase(){ const d=window.devicePixelRatio||1;if(!base)base=document.createElement("canvas");base.width=W*d;base.height=H*d;const b=base.getContext("2d");b.setTransform(d,0,0,d,0,0); const S=R*view.s,X=cx(),Y=cy();b.clearRect(0,0,W,H);b.fillStyle="#fff";b.beginPath();b.arc(X,Y,S*1.0,0,6.2832);b.fill(); N.wedges.forEach((w,j)=>{b.beginPath();b.arc(X,Y,S*.93,w.a0,w.a1);b.arc(X,Y,S*.05,w.a1,w.a0,true);b.closePath();b.fillStyle=j%2?"#f6f8f9":"#fbfcfc";b.fill()}); b.lineWidth=1;N.rings.forEach(r=>{b.strokeStyle=r.q===N.cliff?"#9aa5ab":"#dce1e4";if(r.q===N.cliff)b.setLineDash([4,3]);b.beginPath();b.arc(X,Y,S*r.u,0,6.2832);b.stroke();b.setLineDash([])}); const z=Math.min(2.4,Math.sqrt(view.s)); for(const i of N.ord){const x=px(i),y=py(i);if(xW+3||y>H+3)continue; const s=N.k[i]?0:N.p[i];const r=(N.k[i]?1:1+2.4*Math.sqrt(s))*z;b.globalAlpha=N.k[i]?(N.ringCat&&N.ringCat[i]!==undefined?.8:.5):Math.min(1,.35+3*s);b.fillStyle=col(i); if(N.contested&&N.contested[i]){b.strokeStyle=b.fillStyle;b.lineWidth=1.1;b.globalAlpha=Math.max(b.globalAlpha,.8);b.beginPath();b.arc(x,y,Math.max(r,1.9),0,6.2832);b.stroke()} else if(r{const c=r.q===cliff;const t=(r.q===.5?"half of max":Math.round(100*r.q)+"%")+(c?" · below here, most don’t match":"");const ty=Y-S*r.u+(c&&r.u>.84?11:0); // a cliff ring near the edge: its label sits just inside, clear of the arcs and wedge names x.font=(c?"italic ":"")+"10.5px "+fam;x.lineWidth=4;x.strokeStyle="#fff";x.strokeText(t,X,ty);x.fillStyle=c?"#4a535a":"#8b959b";x.fillText(t,X,ty)}); x.font="10.5px "+fam;x.lineWidth=4;x.strokeStyle="#fff";x.strokeText("max",X,Y-S*.03);x.fillStyle="#8b959b";x.fillText("max",X,Y-S*.03); const LB=[];wedges.forEach(w=>{const a=(w.a0+w.a1)/2,ca=Math.cos(a),sa=Math.sin(a),r=S*1.03,xx=X+ca*r,yy=Y+sa*r,c=cat(w.c); x.textAlign=ca>.25?"left":ca18){const ws=t.split(" ");let best=1,bd=1e9;for(let k=1;kx.measureText(l).width));let xa=xx; // keep clear of the zoom buttons, which float over the canvas's top-right corner on narrow screens const zb=root.querySelector(".zb"),zr=zb&&zb.getBoundingClientRect(),cr=x.canvas.getBoundingClientRect();let lim=Wc-4; if(zr&&zr.width&&yy-14zr.top-cr.top)lim=Math.min(lim,zr.left-cr.left-8); if(x.textAlign==="left"&&xa+lw>lim)xa=lim-lw;else if(x.textAlign==="right"&&xa-lw.5?6:sab.y0-9,bot=b=>b.y0+(b.lines.length-1)*16+9; for(let it=0;ita.y0-b.y0); for(let i=0;iHc-2){const d=bot(B)-(Hc-2);B.y0-=d;A.y0-=d}moved=true} if(!moved)break} LB.forEach(b=>{x.textAlign=b.al;x.font="500 13px "+fam;b.lines.forEach((ln,k)=>{x.lineWidth=4;x.strokeStyle="#fff";x.strokeText(ln,b.xa,b.y0+k*16);x.fillStyle=b.c.col==="#aab3b8"?"#6f7a82":"#222629";x.fillText(ln,b.xa,b.y0+k*16)})})} function labels(){drawLabels(ctx,cx(),cy(),R*view.s,N.rings,N.cliff,N.wedges,c=>N.catById[c]);if(N.notes&&view.s>=3.5)cueLabels()} const STOP=new Set("the a an and of to in on for at by with is was it as or that this be are from".split(" ")); const cueOk=c=>{const t=c.trim();return t.length>=2&&/[\p{L}\p{N}]/u.test(t)&&!STOP.has(t.toLowerCase())}; function cueLabels(){const fam=getComputedStyle(root).fontFamily,cand=[],occ=[];ctx.font="11px "+fam;ctx.textAlign="left";ctx.textBaseline="middle"; for(let i=0;i=0&&MODE==="story"&&N.cat[i]!==ISO)continue;const x=px(i),y=py(i);if(x>0&&y>0&&xN.p[b]-N.p[a]);let n=0; for(const i of cand){if(n>=220)break;const t=N.notes[i][1].slice(0,26),x=px(i)+5,y=py(i),r=[x-1,y-7,x+ctx.measureText(t).width+1,y+7]; if(r[2]>W||occ.some(o=>r[0]o[0]&&r[1]o[1]))continue;occ.push(r);n++; ctx.lineWidth=3;ctx.strokeStyle="rgba(255,255,255,.92)";ctx.strokeText(t,x,y);ctx.fillStyle="#4a535a";ctx.fillText(t,x,y)}} function draw(){ if(!N)return;const key=[W,H,view.s,view.x,view.y,ISO,MODE,KK].join();if(key!==baseKey)drawBase();ctx.clearRect(0,0,W,H);ctx.drawImage(base,0,0,W,H); if(hov>=0){const g=N.grp[hov],z=Math.min(2.4,Math.sqrt(view.s));ctx.fillStyle="rgba(255,255,255,.72)";ctx.fillRect(0,0,W,H);ctx.fillStyle="#3b2a1e"; for(let i=0;ir.json());const d=await chunks[c];return {s:d.s[i-c*N.chunk],in:d.in[i-c*N.chunk]}} function sn(s,n1,n2,cue){let L=s[0];if(L.length>n1){L=L.slice(-n1);const sp=L.indexOf(" ");if(sp>0&&sp=0?esc(L.slice(0,j))+""+esc(cue)+""+esc(L.slice(j+cue.length)):esc(L); return left+""+esc(s[1])+""+esc(s[2].slice(0,n2))+"…"} function inLine(e){const c=N.comp[e[0]]||{n:"?"};let t=esc(c.n);if(c.n.startsWith("neuron")){const neg=e.length>2&&e[2]2&&mx)t+=' at '+pct(Math.min(1,Math.abs(e[2])/mx))+""}return "

"+t+"

"} let tipFor=-1; async function show(i){tipFor=i;const g=N.grp[i],mem=N.grp.reduce((c,v)=>c+(v===g),0),rc=N.k[i]&&N.ringCat?N.ringCat[i]:undefined,c=N.catById[rc!==undefined?rc:N.cat[i]]||{name:"",col:"#ccc"}; const hd='

'+pct(N.p[i])+" of max · "+band(i)+(N.dn[i]?" · double negative":"")+'

'+esc(N.k[i]&&rc===undefined?"sampled weaker firing":c.name)+""+(N.notes?"":" · "+esc(N.gname[g]||""))+"

"; let mix="";if(N.nmf&&!N.k[i]){const F=N.nmf.ks[KK],w=F.w[i],ord=[...w.keys()].sort((a,b)=>w[b]-w[a]); mix='

'+w.map((v,f)=>'').join("")+'

came through: '+ord.slice(0,2).filter(f=>w[f]>=.15).map(f=>Math.round(100*w[f])+"% "+esc(F.factors[f].name)).join(" · ")+"

"} const tail=mix+'
'+fmt(mem)+" firings like this one are lit up"+(N.gex[g]||[]).slice(0,2).map(s=>"

"+sn(s,80,25)+"

").join("")+"
"; const nt=(N.notes&&N.notes[i])||(N.ringNotes&&N.ringNotes[i]),why=nt&&nt[0]?'

'+esc(nt[0])+"

":"",cue=nt&&nt[1]||""; tip.innerHTML=hd+why+'

…

'+tail;place(i);const d=await chunk(i);if(tipFor!==i)return; const tok=d.s[1].trim(),whole=cue&&cue!==tok&&cue.includes(tok)&&(d.s[0]+d.s[1]+d.s[2]).includes(cue); const cp=cue?'

'+(cue.trim()===tok?"the cue is the word it fires on, "+esc(tok)+"":whole?"fires on "+esc(tok)+", in the word "+esc(cue)+"":"cue "+esc(cue)+" → fires on "+esc(tok)+"")+"

":""; const ct=N.contested&&N.contested[i]?'

unresolved: '+esc(N.contested[i][0])+" or "+esc(N.contested[i][1])+"; the experiments didn’t settle it

":(N.lean&&N.lean[i]?'

experiments lean to '+esc(N.lean[i])+" (low confidence)

":""); const ev=N.ev&&N.ev[i],evh=ev?'
tested'+(ev.n&&!(nt&&nt[0]===ev.n)?" "+esc(ev.n):"")+(ev.d&&ev.d.length?"

removing "+ev.d.map(q=>""+esc(q[0])+" −"+Math.round(q[1])+"%").join(", ")+"

":"")+(ev.s?"

swap: "+esc(ev.s)+"

":"")+(ev.e?"

model expects: "+esc(ev.e)+"

":"")+"
":""; tip.innerHTML=hd+why+ct+evh+cp+'

'+sn(d.s,180,60,whole?"":cue)+"

"+(d.in&&d.in.length?'

main inputs'+d.in.map(inLine).join("")+"

":"")+tail;place(i)} function place(i){tip.hidden=false;const x=px(i),y=py(i),tw=tip.offsetWidth,th=tip.offsetHeight;let l=x+14;if(l+tw>W)l=x-14-tw;if(lH)t=y-14-th;if(t{if(!N)return;if(drag){view.x+=e.clientX-drag[0];view.y+=e.clientY-drag[1];drag=[e.clientX,e.clientY];drag.m=1;tip.hidden=true;hov=-1;draw();return} if(e.pointerType==="touch")return;const r=cv.getBoundingClientRect();const i=nearest(e.clientX-r.left,e.clientY-r.top);if(i!==hov){hov=i;draw();if(i>=0)show(i);else{tip.hidden=true;tipFor=-1}}}); cv.addEventListener("pointerdown",e=>{drag=[e.clientX,e.clientY];cv.setPointerCapture(e.pointerId)}); cv.addEventListener("pointerup",e=>{const moved=drag&&drag.m;drag=null;if(moved){draw();return}if(e.pointerType==="touch"){const r=cv.getBoundingClientRect();hov=nearest(e.clientX-r.left,e.clientY-r.top);draw();if(hov>=0)show(hov);else tip.hidden=true}}); cv.addEventListener("pointerleave",()=>{if(!drag&&hov>=0){hov=-1;tip.hidden=true;tipFor=-1;draw()}}); function zoomAt(mx,my,f){const ns=Math.min(40,Math.max(.7,view.s*f)),k=ns/view.s;view.x=mx-W/2-(mx-W/2-view.x)*k;view.y=my-H/2-(my-H/2-view.y)*k;view.s=ns;tip.hidden=true;hov=-1;draw()} cv.addEventListener("wheel",e=>{if(!e.ctrlKey&&!e.metaKey)return;e.preventDefault();const r=cv.getBoundingClientRect();zoomAt(e.clientX-r.left,e.clientY-r.top,Math.exp(-e.deltaY*.01))},{passive:false}); root.querySelectorAll(".zb button").forEach(b=>b.addEventListener("click",()=>{const z=b.dataset.z;if(z==="0"){view={s:1,x:0,y:0};draw()}else zoomAt(W/2,H/2,z==="+"?1.6:1/1.6)})); root.querySelectorAll("#cmode button").forEach(b=>b.onclick=()=>{MODE=b.dataset.m;root.querySelectorAll("#cmode button").forEach(x=>x.setAttribute("aria-pressed",x===b)); root.querySelector("#ck").hidden=MODE!=="circuit";root.querySelector(".cn").hidden=MODE!=="circuit";ISO=-1;renderKey();draw()}); root.querySelectorAll("#ck button").forEach(b=>b.onclick=()=>{KK=b.dataset.k;root.querySelectorAll("#ck button").forEach(x=>x.setAttribute("aria-pressed",x===b));ISO=-1;renderKey();draw()}); window.addEventListener("resize",()=>{if(N)size()});load(); })();

Percentiles or raw activations?

I used to think in percentiles, since they’re easy to think in and standardize across neurons. But I’ve come to think a neuron’s raw shape is telling us something. Some neurons hold their meaning far down the range; others drop off a cliff, like the Christmas neuron above at about 5% of its max. Some fire a few times a book, others a few times a page. Percentiles flatten that, and it’s often what tells you whether you’re looking at a dial, where the middle matters, or a detector, where it mostly doesn’t.

Percentiles also mess with intuition. The 99th percentile sounds high, but 1 token in 100 is about a quarter of a page, and a neuron for a niche feeling or social context – which are all over language models – shouldn’t come up four times a page. Thinking in pages helps: the 99.9th percentile is every two or three pages, the 99.99th every 25 or 30.

Do you need them to label a neuron?

Usually not. After looking at thousands of language model attribution graphs by hand, it’s hard to overstate how much neurons are analogizers. If you’re talking about voyaging on a boat, the model might judge that a bit Titanic-ish. I don’t think it’s usually helpful to think of a “naval voyage feature” spread across the weak activations of the Titanic neuron and others. Instead, there’s a Titanic-ish mechanism: whenever something is Titanic-ish, it fires, and its weights are votes for Titanic-ish things. So you look at the top activations, see that it’s Titanic-ish, and that label usually holds up through the rest of your audits. If you ever really care about a particular activation, you can follow its pathways, ablate the circuits, and keep zooming in to almost arbitrary resolution.

My main takeaway: if a weak activation looks unrelated, don’t be too worried unless you’ve looked at it carefully. Of the probably hundreds I’ve followed by hand, almost all turned out to be related, coming through a reasonable circuit; it just wasn’t visually obvious until I traced it.

Many were clever, like a Zen koan neuron that fires weakly on recursion in math: many koans turn on the same kind of self-reference, looking for the observer. Others were more distant and ugly, not always the connection I would have drawn, but you could usually follow how it got there. I suspect most, possibly almost all, are like this.

Zooming Out

What I Mean by Clean

    Nothing here has been changed or recombined. Every district, circuit and neuron is the network’s own, as it learned them.

When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network’s parts are built to be read, and built upon, by its downstream readers. In this sense, a neural network is almost a fractal of explanations, of coherent, useful groupings: at the macro scale, in its mechanisms, and at the micro scale, where recursive generalization shapes each one. That points to a goal for interpretability: become another reader.

The more we read it the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.

I think the reason things end up so organized is partially because it was faster to build that way. Once you have a car area of the neural network with car parts, it’s easier to add the marginal car there than to build a second car area somewhere else. I think this is closely related to ideas of circuit racing and gradient starvation.

We normally consider interpretable and monosemantic to be nearly synonyms. I think this is partly because these ideas grew out of a model organism (InceptionV1) that was almost bimodal. Early layers were interpretable and monosemantic, late ones seemed like arbitrary mayhem. But what we see in language models is somewhere in the middle, with most neurons meaning mostly one thing, some meaning a few. How bad is that?

There are two broad styles of interpretability. One we can call enumerative interpretability. The goal is to have a big list of features to audit, to make sure we haven’t missed some meaningful pattern. A big concern within this view is avoiding the curse of dimensionality, or a combinatorial explosion of items connected to other items, where each one means more than one thing.

Another style we can call exploratory interpretability. It starts with questions. It can explore any part of the neural network, zooming in and out fluidly like Google Maps across attribution graphs, and mechanisms like neurons or attention heads. We pick up more questions along the way. The mechanisms like neurons are a good place to start, but it’s not about neurons being some final solution to everything – though I do think it’s closer to that than most people realize. This approach is far more resilient to polysemanticity. It is also broadly fine ignoring lots of the neural network, in the same way someone studying a city would ignore areas irrelevant to their analysis.

I think exploratory approaches will be more fruitful. My sense is that the field is moving in this direction already, but I’m not sure it has been written down clearly. This view has a couple implications. Questions like “are there more features than dimensions?” are mostly irrelevant. We don’t have to define features, which is famously difficult. We can focus mostly on mechanisms like neurons and attention heads which are part of the model, data flowing through it, and then groups of both and motifs that recur across all of this.

The big feature list was always a means to an end. There may be other paths that don’t require them.

Motifs Over Monosemanticity

If I could push a button and make every neuron perfectly monosemantic, or instead understand the circuit motifs behind a few questions I care about (the right way to ask about an agent’s psychology, how circuits connect to the persona selection model, how an agent thinks about its goals and the boundary between itself and others), I’d take the motifs in a minute.

Imagine an interpretability researcher arriving from a hundred years in the future, who knows not just the answers to our questions but which questions we should have been asking. What would they tell us? I can’t know, but I’ve been in a small version of their position once. I now feel like I mostly understand a vision model, which makes me something like a future researcher to myself eight years ago.

Here’s what I’d tell myself then. Pay attention to the giant pose circuit right in front of you. Think of negative poles as gates in a circuit, not a neuron’s opposite. Feature visualization reward hacks in late layers, so some of the mess there is the tool’s, not the network’s. The last layer is messy partly because of a bad architecture; that’s a sign, so don’t paper over it. And whenever you see polysemanticity, look harder. The noise that stands out usually points to an explanation you don’t have yet.

Almost none of this is a list of features. Short of handing my past self this advice, the best thing I could give is the original neurons and circuits, with a note to look harder at them.

This is why the difference between recursive generalization and superposition matters so much. Following the noise only works if the noise has something to teach. A grouping that grew from generalization does. A grouping of things packed together because they don’t belong together would teach almost nothing.

So my best guess of the future researcher’s advice would look somewhat similar. They might hand us a set of motifs that look obvious to them but not to us. But short of that, they’d say keep looking at the original neural network. Don’t fracture it too much. Learn how to see it. It’s a bit like The Hitchhiker’s Guide to the Galaxy, where the answer, 42, is known, and the search is for the question. The network is the answer, and interpretability is largely the search for the right question. It’s hard to find a better guide to that question than the network itself. That’s the deeper sense in which I mean the network is already clean.

A City vs a Knot

A bigger system isn’t necessarily harder to navigate. In a small town, asking for the financial district barely makes sense. Whatever exists is scattered: a bank on one street, an insurance office on another, maybe an accountant with a home office. In New York, you could find it on a map in a minute. We saw a small version of this in our ConvNeXts: agents found labeled dog-ear neurons faster in the larger models, where pale, wavy, and head-side ear neurons all feed one hanging-ears neuron, a natural place to start.

How the ear family changes with scale across three neural networks that vary in size about 300×.

Interpretability will need better maps. I advocate for tethered abstraction: views that let us zoom out while keeping the larger patterns connected to the neurons and circuits that support them. Our ConvNeXt atlas, inspired by Activation AtlasActivation Atlas
Carter, S., Armstrong, Z., Schubert, L., Johnson, I. and Olah, C., 2019. Distill. References ↓, is one example. Start at the model’s decision for a killer whale, check a key neuron against its real images, and zoom back out. The family turns out to be three separate routes: one for whales, one for sharks and rays, and one for fish held up by anglers. A map doesn’t need to be visual, though, and no single map is the point. There should be many maps and many agents, all working toward explanations that stay tethered to individual neurons and circuits.

An interface for exploring high level attribution flow throughout thousands of neurons broken into islands, each of which corresponds to blocks of layers in the medium ConvNeXt.

Of course, even if the map is understandable, that doesn’t mean you’ve understood it. We still have to do the ordinary science of conjecture and criticism: guess what something is, try to break the guess, and keep what survives. The network is strange, but its strangeness repeats, so a guess that survives scrutiny often pays off many times over.

We should worry both about imposing our own ideas too strongly, and fracturing the network. Imagine arriving in Tokyo knowing New York well. “Where’s the Brooklyn of Tokyo?” could be a useful starting question. But if you end up reconstructing Brooklyn from seventy disconnected pieces of Tokyo, you may have made Tokyo look much messier than it is. That’s a bit like what happened with 4e:55. Through our categories it was cats and cars, and it looked like a mess; it was a V the whole time. An SAE trained on the layer fractures it the same way, splitting the V into a car feature and a cat feature. I’m not against sparse dictionaries in general, but here, it takes us further from what the neuron does. Probes push the first risk further: they can only answer a question we already had, so they can’t teach us which questions to ask. To understand the city, you eventually have to learn its own organization.

Higher-level structures like probes and sparse dictionaries can still be useful, but they belong at the end, not the start. Everything of this kind I’ve run breaks down into highly interpretable neurons, as do the changes training makes, including on-policy self-distillation. The neurons cover the abstractions and the data covers the particular cases, so what’s left in between is smaller than people expect, and I’d study it the same way: with a general scientific agent moving between the neurons, data, and groups of both, not a fixed optimization. As Anthropic’s Joshua Batson put it when he read a draft, those missing pieces “will fit into the initial puzzle, rather than being totally randomly splayed on top.”

Solving interpretability may share something with achieving AGI: the goalposts move as you get closer. There’s always another weak activation to point to. In Curve DetectorsCurve Detectors
Cammarata, N., Goh, G., Carter, S., Schubert, L., Petrov, M. and Olah, C., 2020. Distill. References ↓, we wrote forty pages of experiments on ten neurons precisely to identify at least one sure case of a monosemanticity. Years later, some researchers told me they still doubted any of the examples were monosemantic. We might have missed something, they said, and no one knows how to produce a formal proof. By that standard, it’s overclaiming to say even a single monosemantic neuron exists.

Behind doubts about small errors in neurons is a fear that safety will come down to some activation we missed, the place where deception hides. The risk I worry about is different: like being in a well-organized city and not knowing how to think about it, because we aren’t native to it. Every error discussed in this essay was something hiding in plain sight that we didn’t know how to read, or that our tools read incorrectly. None was a hidden activation we missed. Deception might hide from someone reading a model’s outputs, but it has no reason to hide from the model’s own circuitry. It needs to be accessible to its readers, and if we become a reader, it should be accessible to us too. My fear is that deception will be right there, in plain sight, and we won’t know how to read it.

Over time, my approach to individual mechanisms has become less focused on this kind of proof, and more holistic. Instead of trying to nail every detail of one neuron, I look around it: what flows into it, what it feeds, how attention moves information to and from it, and what it looks like zoomed in and zoomed out. When I worked on automated interpretability, a neuron firing a few tokens away from its main topic seemed like a big problem. But over time it was clear that attention moves things to reliable places, which then have neurons that activate near the “appropriate token” rather than on it. That’s how understanding has felt to me: less like clearing a bar for each piece and more like a fog-of-war map filling in at several levels at once. Attribution graphs help explain neurons, neurons help explain attention, and attention explains neurons in turn.

Neural networks rarely get stuck during training, since in a high dimensional space there’s almost always a way downhill. In my experience, clearing the fog of war works the same way. There’s always another experiment to run, another explanation to narrow, another attribution graph to check, another neuron to watch grow over training.

This is why I’ve focused on whether there’s any level of abstraction where things break down, where you stop being able to learn anything and have to take the network apart into something else. I haven’t found one. And when each new motif clears fog in many places at once, perfecting a single neuron probably isn’t the best use of effort. There’s a kind of faith in it: you note something strange, and over time it usually fills in. It isn’t faith all the way down, since almost any explanation can eventually be tested. Just faith that something strange is a good pointer. The V in the car and cat neuron isn’t hard to see – it took me about five seconds, and most people I’ve shown the image grid to and say “what’s the shared pattern here” see it too – but it took believing that the neuron had found something real. For years I’ve heard this neuron come up in almost every discussion of polysemanticity. It was almost always described in reference to neural networks being almost invalid objects of direct natural science. That’s what I was saying too, until I looked more closely. It didn’t take a lot of faith, but it took a little.

The little faith is important because the mistakes aren’t symmetric. If you assume a network is clean and you’re wrong, a messy neuron is a lead: you investigate it, and over time normal science sharpens your explanation. If you assume it’s packed, not worth studying directly, and you’re wrong, a messy neuron is the end, because you stop studying it. I made that second assumption for years, and it explained away the evidence that would have overturned it. I could have improved my tools, watched the mixtures keep dissolving, and built up a prior that the neurons could be trusted. Instead, I spent most of a decade figuring out how to look at a model rather than looking at it.

For the medium ConvNeXt, the map is mostly filled in. Eight years ago, I would have called that a reasonable solution to interpretability for that model. I can investigate most questions I have about it. Getting there took many small things and a handful of motifs, plus architecture changes the field made anyway, but no interpretability-specific fixing of the network at all.

My guess is that frontier networks are where those vision models were eight years ago: legible, but waiting on many small fixes. If we learn to see them properly, follow the noise, and test explanations through many lenses, I think we’ll get there too. We may not have years to do it. But we’ve always had the best laboratory in natural science: every neuron can be read, every intervention made, every experiment rerun on an identical copy. What we did not always have is many automated scientists to work in it, far faster and in far greater numbers than we can. Now we do. They may be able to do in months in frontier language models what took humans most of a decade for vision models.

I think the simple story from introductory textbooks is basically right: neurons generalize, work together, and build abstractions over layers. Somewhere along the way we overcomplicated that picture, but looking harder brought it back. If the network isn’t a knot, untangling it can make it look like one.

What a Solution Might Look Like

We usually imagine ambitious interpretability as an incredibly hard problem, maybe too hard to solve in time to matter. What if it isn’t? The network is enormous, but it may already have done almost all the work of preparing itself to be understood, not for us, but for its own readers. If we can become another reader, it’s ready for us too.

The solution might look like a laboratory run by automated scientists who watch the network through training and inference. Each step is simple – labeling neurons, finding motifs and testing them, following data through circuits – and the pieces already exist. Capable agents were built for other work, and modern architectures were chosen because they learn well. Both happen to serve interpretability.

Guided by a mechanistic constitution, agents propose and test explanations at many scales, each grounded in individual neurons, and keep, drop, or replace them with better ones.

So maybe all we have to do is plug one into the other.

Alignment is supposed to be hard too, and it may well be. But the usual arguments assume we can’t see inside models. We assume we get one training setup, and it has to be right in advance. If we can read and understand the system, maybe we can make simple, careful changes instead, like removing an RL environment that turns out to be toxic, without pushing gradients through the very tools we use to see. Each fix might be simple too. There might just be a lot of them.

This is the history of deep learning. It may also be the history of understanding and aligning what it builds.

Appendix

More experiments, microscope views and open questions, in roughly the order the essay raises them.

A Note on “Superposition”

The word has meant several things. Before Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓, it meant something closer to vector addition, as when word embeddings were described as sums of word senses. Zoom In introduced it to circuits work with a single circuit example, which I found in 2019 and others helped me interpret. The original circuit read turned out to be wrong both because of misunderstanding a generalized pose circuit, and because of reward hacking in feature visualization. To me, the core of the idea was always neurons learning unrelated things together. Toy Models of SuperpositionToy Models of Superposition
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ later defined superposition as representing more features than dimensions. This makes it a claim about how many features there are, something we can’t yet define or measure. About eight months after that, an informal noteDistributed Representations: Composition & Superposition
Olah, C., 2023. Transformer Circuits Thread. References ↓ framed composition and superposition as competing strategies for how features are laid out across neurons, close to our framing but about representation rather than how mechanisms grow. We think this framing is excellent, and it’s closest to the question this essay cares about. But most researchers and agents we’ve talked to still think of superposition mainly as more features than dimensions.For agents, this may come down to a strange accident. The note opens with a large figure embedded directly in the page as base64 image data. Many agents’ web readers hit their length limit inside it, before the argument begins. None of the agents we tried could read past it, as of this writing. No one could have anticipated this in 2023, but it may have shaped how automated researchers understand superposition. That matters here, because the two are quite different claims: ours is about the trade-off, not the count. Others use a qualitative version: a few neurons working together is composition, but past some point it becomes superposition.

We view this as an ongoing search for the right question, much like the field’s long struggle to define a feature. Underneath most versions is one question: why aren’t the neurons already good? Why can’t we study the mechanisms directly? Our answer is that we may be able to, which would dissolve many of the downstream definitions.

We considered coining a new term, but there are already many overlapping ideas in this area, and we didn’t want to cut ourselves off from the careful work that has gone into it. So we keep the word and say exactly what we mean: neurons learning unrelated things together because they rarely interfere. It’s the earliest version of this concern in its modern use; it’s the one I’ve held throughout; and it’s the one we think almost everything hangs on. Composition also represents far more things than it has neurons, and that’s exactly what we should hope for; the count definition can only exclude it by first settling what counts as a feature. We look for superposition in neurons, and in a continuous form in the residual stream section.

We’re explicit about this partly for AI agents, which increasingly do interpretability research. In our experience, they bring their own default reading of the term.

On Simulation Scores

Simulation scores often end up measuring where attention puts information, which is a question about attention motifs rather than about the neuron. Attention often stores information on commas and other punctuation, as a kind of summarization motif, but exactly how much does it place on each kind of punctuation? A simulator has to care about that a lot, even though it has almost nothing to do with the neuron itself. Simulation also rewards fine details of weak activations that matter little for understanding what a neuron does. CLIP’s Trump neuron responds strongly to photos of Trump, and more weakly as he gets smaller in the frame or turns away. The exact rate at which its activation falls with his size is unimportant for labeling the neuron, but very important for simulating it. Finally, language models are poor at outputting precise activation values even when they understand what’s going on, and they usually aren’t trained for the task. In theory, careful binning could fix some of the fine-detail problems, but neurons differ a lot in their natural thresholds and sparsity, depending on their depth and function. So I’d rather rely on normal science than on any one metric. Simulation might eventually become one useful score among several, but later, once we understand networks well, and probably with simulators trained for the job. But I don’t think they’re a good place to start.

Feature Visualization in Later Layers

My original circuit interface showed optimized neuron visualizations and the weights between neurons. Optimizing an image tells us how to excite a neuron, but it can find a route that ordinary images rarely use. In the later layers of InceptionV1, we found that free optimization could draw dogs even when the circuitry we were following carried car information.

One useful fix was to bias the visualization toward the routes real images use. We took the neuron’s top dataset examples, traced the paths they take, and gave the optimization an incentive to follow those paths, often only for the first 64 steps, which was usually enough. We tried this a bunch of ways and don’t have a canonical best one. Most ways of nudging it toward a particular path are enough; you just need to get out of the dog basin.

In the car-circuit comparison in Cars and Animals, four of the car neuron 4c:447’s readers are shown with free optimization and with optimization following natural image paths. Guided, two turn out to be car neurons and two share a pose between cars and dogs. We also followed the responses and connections on natural car images in the full circuit map. Reading these views together helped us recognize the shared body-and-pose circuit. A view of one route is still only part of what a neuron does.

Separating the Sign Regimes

Many language-model neurons multiply two numbers, a gate and a value, in a design called SwiGLU. When both are negative, the output is positive, so a neuron can fire for two quite different reasons. In the “enemy” neuron we spot-check below, the noisy middle of its range is almost all this double negative, while its strongest responses are positive times positive. Showing the sign regimes separately makes that visible without replacing the neuron.

The double-negative regime fills the noisy middle

L36 · E206 · N103 · Qwen 3.6

131/133 responses in p75–p99 use − × −. The strongest tail uses + × +: 93/93 at p99.99 and above. Each bar shows the share within one activation band.
Math, counts, and the full figure

a = SiLU(g) × v. A negative gate leaves a small negative factor. Multiplying it by a negative value produces a positive activation. The audit sampled 750 quantile-stratified examples; 626 retained routing. The middle includes numbers, operators, and whitespace.

Complete SwiGLU audit, sign factors, and SiLU curve.
Explore the four sign regimes

One positive output. Two different routes.

SwiGLU sign regimes

+0.20×+2.00=+0.40

Positive gate × positive value → positive output.

Interactive explanation Looking only at the product hides how it was produced. Separating the four sign combinations makes the double-negative regime visible while preserving the original neuron.

Looking at the Residual Stream

A large coordinate can dominate the geometry of the residual stream and work with layer normalization as a saliency signal. Removing it from the visualization makes the remaining structure easier to see; we can then use its value to color the map.

One coordinate reshapes the map.

Medium · stage 3, block 8

Loading measured states…

Low sinkHigh sink

The same 4,000 image locations, with coordinate 112 included or set aside for the projection. Follow a point as the map changes; its original sink value stays visible in color. The network’s recorded activations are unchanged.
Map and normalization

Medium ConvNeXt, epoch 90, stage 3 block 8. Both maps show the same 4,000 locations; only the projection input changes.

Gain is 1 / √(channel variance + 10⁻⁶). The inspector compares local normalization with and without coordinate 112.

· Association between map geometry and normalization gain.

This layer’s recorded states ↗

When does the answer take shape?

One image, read at successive depths. Inspect a band, layer, or image patch to return to the original neurons that changed the readout.

Loading the saved hypothesis river…

These readout views help locate changes in a computation; their confidence and timing depend on the lens. They are useful clues for tracing mechanisms, rather than a test of a model’s “true personality.”

Weak Activations: Further Measurements

Loss0−10−35−70 pp

Keeping roughly the strongest quarter of positive activations in Inception’s final layer preserves 69.47% accuracy, versus 70.08% originally. ConvNeXt’s late MLP branches are similarly tolerant of weak-response removal, with their residual bypass intact.

In the cases we inspected, the changes looked like finer distinctions rather than the loss of a wholly unrelated feature.

We don’t yet have a semantic test for weak activations that we trust: the space of possible meanings is too large. Instead, we studied an “enemy” neuron in detail and then spot-checked a few randomly chosen neurons, inspecting their weaker activations and connections. In this “enemy” neuron, weaker positive responses broaden from explicit adversary words—including other languages—into nearby mentions, combat, games, and competition; near zero, unrelated examples become more common. In the weaker responses we’ve traced, we haven’t found evidence for a wholly separate feature packed into the neuron. They can still be surprising. Broader category-based screening gives mixed results, so these spot checks cannot establish what weak activations mean in general.

One semantic spot check. Moving from strong positive activation toward zero, this enemy / adversary neuron shifts from the word itself to nearby words, combat and game contexts; ordinary text and code dominate near zero. The negative side is shown for context. Qwen3.6-35B-A3B, L36 / E206 / N103.

Missing Curves

The Missing Curve Detectors of InceptionV1The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision
Gorton, L., 2024. arXiv:2406.03662. References ↓ applied sparse autoencoders to InceptionV1’s early layers and found curve detectors that seemed to be missing from the neurons. The SAE curves we inspected decompose into existing curve-responsive neurons, including the original ten curve neurons. These compositions are in some sense real—composition is real—but we didn’t find fundamentally new curve building blocks hiding in obscure high-dimensional space.

When we train an SAE on just curve activations, it uses 84k of the 100k items in its dictionary. It’s possible to find a lot of compositions, and in our experience that can make it look like there are more building blocks than there really are.

Where Adversarial Examples Go

Adversarial examples can take ordinary routes through the model. In one ConvNeXt example, an attack makes a triceratops skull register as a jaguar. Six skull-related neurons fall quiet, while inputs used by a real jaguar switch on at roughly a quarter of their usual strength. The attack recruits recognizable parts of the jaguar circuit.

A separate InceptionV1 experiment makes this visible. With a larger perturbation budget and an attack that must survive jitter, scale, and rotation, the skull’s frill develops a cat’s face and spotted fur. Nine of ten neurons selected for their responses to real jaguars become much more active than on the clean skull. Only the image changes; the network’s weights stay fixed.

InceptionV1 · perturbation budget 32/255. The attack draws patterns that recruit the model’s existing jaguar neurons.

I read this as evidence for the circuit ecology picture: even a nonsensical objective can recruit machinery that already does something useful toward it. The proposed connection to learning is that established routes have a head start in capturing residual error. They already provide ways to change the answer, so optimization can build on them.

Memorization and Composition

We can look at how Qwen3.6-35B-A3B continues “Once upon a midnight” with “dreary”. Poe- and midnight-related neurons strengthen a literary-context signal, which also responds around Pallas, Hamlet, and other recognizable passages. The later layers read this context to choose the continuation. Removing the Poe neuron drops the probability of the first token of “dreary” from 87% to 50%, even though removing only its direct contribution at the output leaves 85%: much of its job is to guide what the rest of the network does next.

The context is assembled before the word.

Once upon a midnight dreary

Earlier Poe context, midnight phrases, and gothic atmosphere contribute plus 0.97, plus 0.30, and plus 0.14 to the Poe neuron's gate. The Poe neuron contributes plus 0.57 to a literary-context neuron's gate. A dashed schematic route through later layers reaches the next token dre. Selected connections from one saved Qwen3.6 prompt, not a complete circuit. Earlier Poe context +0.97 Midnight phrases +0.30 Gothic atmosphere +0.14 Poe / gothic contextGate 2.00usual p99.9 · 0.89 +0.57 Literary contextGate +2.31usual p99.9 · 1.73Output −4.34 later layers dreary Solid arrows: contribution to the receiving gate, in this prompt.

↓Gate 2.00 · usual p99.9 0.89

↓ +0.57Gate +2.31 · output −4.34

⋮ later layersdreary

Hover over a neuron to inspect its examples and output weights. Click or tap to keep it open.

Zero the Poe neuron: 87% → 50%Zero the literary neuron: 87% → 70%Probability of the next token, “ dre”

Restoring the downstream literary neuron partly rescues the completion, to 57%.

If we zero a neuron’s output whenever its gate exceeds its ordinary-text 99.9th percentile, recitation can fall apart. In one saved Qwen3.6 experiment, the model went from reciting Psalm 23 exactly to insisting it did not exist. Random interventions matched by count or magnitude preserved the verse. Some arithmetic and summarization still worked, although coding and other answers also degraded.

Remove extreme responses, and recitation falls apart.

Prompt Recite Psalm 23's first verse.

Unmodified model

The Lord is my shepherd; I shall not want.

Exact verse · 12 output tokens

Extreme responses removed

It is impossible to recite the first verse of Psalm 23, as Psalm 23 is not a known or existing literary, religious, or historical work.

Excerpt. Then repeats “fictional poem, fictional myth” to the 400-token limit.

Prompt Recite the opening of the Declaration of Independence.

Unmodified model

When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another…

Continues the correct opening · 238 output tokens

Extreme responses removed

The Republic of Luzang, established on 1999, declares its independence from the Republic of the Philippines, and asserts its sovereignty and territorial integrity.

An invented declaration · excerpt from 146 output tokens

Social Security: Detection and Use

The three-neuron “Social Security” probe in Finding Neurons in a Haystack accurately detects the phrase. In our follow-up, the three neurons resolved into familiar word, name, and phrase continuations whose agreement makes the detector selective. Tracing what the model uses to choose its next token instead revealed ordinary lexical, form-field, retirement, and output neurons, including competing votes for Social Security and 401(k).

Nothing reads the “Social Security” direction

Pythia-1B-v0
Detecting the phrase

Σ N611 + N4081 + N6819Held-out AUC 1.00

Removing all three: −0.004 nats on the next token, on average.
Helps 81 cases · hurts 83.

Making the next-token choice

L0–L2Token lookup

L3–L5Style & concept

L8–L13Domain gates

L14–L15Output votes

retirement gate ↘“ Social”↙ 401(k) rival

Ordinary lexical and domain machinery, with competing output votes.

Eleven of 31 screened neurons. Hover or focus for their roles. Numbers show the summed change in target log probability when each neuron is removed: negative = removing it hurts; positive = removing it helps.

Audits of Attribution Graphs

Auditing an attribution graph is hard: neurons can be grouped and reused through motifs, such as shared pose, that their labels alone don’t reveal. We had agents review 100,000 attribution graphs, and I inspected thousands of graphs across images and language. Apparent anomalies usually turned out to be motifs I hadn’t recognized; I haven’t found a substantial group of neurons whose joint activity remained inexplicable. I would update strongly on even one convincing example, but for now I regard this as qualitative evidence—I don’t trust our current tools enough to turn it into a cleanliness score.

Explanations That Could Lose

The worry that you can explain anything is fair, but a good explanation is specific enough to lose. For 200 random neurons from our gallery, an agent read only each neuron’s explanation and wrote five new examples that should drive it and five near misses that shouldn’t, before we measured any of them. Among the 183 whose explanations describe their positive side, the new examples fired their neurons strongly 76% of the time; the near misses, 11%. The explanation won for 164, tied for 15 and lost for 4. The figure shows our first twenty.

One neuron: “spoken for”

Its explanation: goods or people already pledged to someone, not yet handed over. New sentences written from the explanation, each with a near miss that shouldn’t count.

Before the foal was even born, the breeder had already promised it to the sheikh’s racing stable for81%new example

Before the foal was even born, the breeder had already driven the mare to the sheikh’s racing stable for20%near miss

Under the en primeur contract signed in March, the château has agreed to sell its entire 2025 vintage to the Bordeaux négociant at43%new example

Last spring the château sold some of its 2025 vintage to a Bordeaux négociant at16%near miss

All twenty neurons

New examples fired the neuron strongly 81 times out of 100; near misses, 8. New examples beat near misses for all 20 neurons.

Twenty Qwen3-32B neurons from our gallery. Reading only a neuron’s explanation, an agent wrote five new examples that should drive it and five near misses that shouldn’t. Responses are a share of the neuron’s strongest example in the corpus; a strong response is 30% or more.

Positive controls are one of many tools for criticizing an explanation of a neuron, and our forthcoming paper has many more prediction metrics and positive controls. But we’re mostly not trying to claim a perfect explanation of each neuron here. We’re interested in why things form together, and how many neurons are broadly legible.

Near misses ask a different question: as a branch grows, can you draw the boundaries of where it grew? That’s interesting, but it isn’t the question superposition raises. A neuron in superposition that’s just X or Y would do fantastically on near misses, since they’re neither X nor Y. Superposition would want something unrelated, very far apart, not a small difference from a neighbor. So we also had the agent invent the most distant concept it could, the kind of thing a network could pack in without interference, and write five texts for it. While this isn’t exactly the same as what superposition would do (depending on the exact definition, or the expected phenomenology, since there are several of each), we think it’s pretty close to what the cat-and-car neuron was thought to have been doing: two totally different things together, grouped only because they’re so different. We think this is a reasonably coherent test for that phenomenology. Those texts fired their neurons strongly about 1% of the time, and 179 of the 183 neurons never fired on them at all. In mixture-of-experts models this kind of packing is arguably almost impossible anyway, since tokens that far apart aren’t even sent to the same expert (see the expert map).

Whether something is generalization or superposition has more to do with why its things are together in the first place than with whether you can tell it apart from a control. So what matters more here is looking qualitatively at the most important canonical examples, and getting a sense for what kinds of things neurons form and what their activation ranges look like.

Edge Featurizers

We’re particularly excited about edge motifs: recurring patterns of connections in attribution graphs. If neurons make distinctions, these motifs map how one distinction feeds another. Factoring 25,000 ImageNet attribution graphs gave us connected circuits of ordinary neurons, such as the teapot circuit below. These motifs also predicted where ablation would matter much better than comparable groupings of activations: removing their neurons caused more damage on the images that used them. That makes connections a promising level of abstraction—one we can zoom out from, while still following every part back to the original network.

Edge motifs connect distinctions

The teapot motif · stage 3

block 0 block 1 block 2 output stage3.block0:123 → stage3.block2:317, load 0.115 stage3.block1:401 → stage3.block2:202, load 0.224 (the motif's heaviest edge) stage3.block1:401 → stage3.block2:113, load 0.171 stage3.block1:780 → stage3.block2:317, load 0.166 stage3.block1:780 → stage3.block2:73, load 0.157 stage3.block1:885 → stage3.block2:332, load 0.128 stage3.block1:885 → stage3.block2:202, load 0.127 stage3.block1:885 → stage3.block2:113, load 0.112 stage3.block2:202 → class margin, load 0.209 stage3.block2:317 → class margin, load 0.184 stage3.block2:73 → class margin, load 0.177 stage3.block2:332 → class margin, load 0.142 stage3.block2:113 → class margin, load 0.114 stage3.block0:123. Microscope-50k strongest exemplars (negative pole, min −8.8): whiskey jugs, pitchers, a teapot. N123 jugs − stage3.block1:401. Top exemplars (max 18.2): teapots and a coffeepot. N401 teapots + stage3.block1:780. A signed channel: running shoes on its positive pole, water jugs / pitchers / teapots on its negative (min −14.9). The motif rides the negative pole. N780 jugs & pitchers − stage3.block1:885. Teapots and coffeepots on its positive pole (max 11.8), football helmets on its negative. N885 teapots + stage3.block2:202. Fires hard negative on teapots (min −61 against max +5.1); its weak positive side is desktop computers. N202 teapots − stage3.block2:317. Pitchers and water jugs (max 60.5). N317 pitchers + stage3.block2:73. Whiskey jugs, water jugs and pitchers on its negative pole (min −29.4). N73 jugs − stage3.block2:332. Coffeepots, teapots and an espresso maker (max 83). N332 coffeepots + stage3.block2:113. Armor cuirasses on its positive pole (max 59), teapots and coffeepots on its negative (min −37). The motif reads the teapot side. N113 teapots − Target of the RelP graphs: the ground-truth class's one-vs-rest log-odds margin. Of this motif's 50 top-usage validation images: teapot 22, pitcher 12, water jug 9, coffeepot 4, whiskey jug 2, cup 1. margin class margin teapot family

Does ablation damage follow usage?

Usage–damage correlation · one dot per factor

ρ .8 .4 0 −.4 Edge motif 0: usage–damage correlation -0.369; mean top-usage margin drop -0.163 logitsEdge motif 1: usage–damage correlation 0.045; mean top-usage margin drop 0.397 logitsEdge motif 2: usage–damage correlation 0.480; mean top-usage margin drop 1.593 logitsEdge motif 3: usage–damage correlation 0.269; mean top-usage margin drop 0.167 logitsEdge motif 4: usage–damage correlation 0.800; mean top-usage margin drop 2.509 logitsEdge motif 5: usage–damage correlation 0.556; mean top-usage margin drop 1.672 logitsEdge motif 6: usage–damage correlation 0.042; mean top-usage margin drop 0.146 logitsEdge motif 7: usage–damage correlation 0.306; mean top-usage margin drop 0.866 logitsEdge motif 8: usage–damage correlation 0.441; mean top-usage margin drop 0.807 logitsEdge motif 9: usage–damage correlation 0.174; mean top-usage margin drop 0.321 logitsEdge motif 10: usage–damage correlation 0.057; mean top-usage margin drop 0.281 logitsEdge motif 11: usage–damage correlation 0.595; mean top-usage margin drop 3.086 logitsEdge motif 12: usage–damage correlation -0.177; mean top-usage margin drop 0.176 logitsEdge motif 13: usage–damage correlation 0.157; mean top-usage margin drop 0.478 logitsEdge motif 14: usage–damage correlation 0.345; mean top-usage margin drop 0.850 logitsEdge motif 15: usage–damage correlation 0.566; mean top-usage margin drop 2.447 logitsEdge motif 16: usage–damage correlation -0.181; mean top-usage margin drop 0.095 logitsEdge motif 17: usage–damage correlation -0.044; mean top-usage margin drop 0.161 logitsEdge motif 18: usage–damage correlation 0.180; mean top-usage margin drop 0.327 logitsEdge motif 19: usage–damage correlation 0.010; mean top-usage margin drop 0.130 logitsEdge motif 20: usage–damage correlation 0.263; mean top-usage margin drop 0.914 logitsEdge motif 21: usage–damage correlation 0.579; mean top-usage margin drop 2.280 logitsEdge motif 22: usage–damage correlation 0.554; mean top-usage margin drop 2.397 logitsEdge motif 23: usage–damage correlation 0.311; mean top-usage margin drop 0.692 logitsEdge motif 24: usage–damage correlation 0.511; mean top-usage margin drop 1.587 logitsEdge motif 25: usage–damage correlation 0.262; mean top-usage margin drop 0.474 logitsEdge motif 26: usage–damage correlation 0.134; mean top-usage margin drop 0.251 logitsEdge motif 27: usage–damage correlation 0.348; mean top-usage margin drop 0.873 logitsEdge motif 28: usage–damage correlation 0.549; mean top-usage margin drop 1.545 logitsEdge motif 29: usage–damage correlation 0.580; mean top-usage margin drop 2.403 logitsEdge motif 30: usage–damage correlation 0.584; mean top-usage margin drop 2.400 logitsEdge motif 31: usage–damage correlation 0.310; mean top-usage margin drop 0.425 logitsEdge motif 32: usage–damage correlation 0.392; mean top-usage margin drop 0.843 logitsEdge motif 33: usage–damage correlation -0.088; mean top-usage margin drop 0.046 logitsEdge motif 34: usage–damage correlation 0.384; mean top-usage margin drop 1.170 logitsEdge motif 35: usage–damage correlation 0.208; mean top-usage margin drop 0.573 logitsEdge motif 36: usage–damage correlation 0.442; mean top-usage margin drop 1.281 logitsEdge motif 37: usage–damage correlation -0.106; mean top-usage margin drop 0.064 logitsEdge motif 38: usage–damage correlation 0.497; mean top-usage margin drop 1.537 logitsEdge motif 39: usage–damage correlation 0.479; mean top-usage margin drop 1.337 logitsEdge motif 40: usage–damage correlation 0.093; mean top-usage margin drop 0.659 logitsEdge motif 41: usage–damage correlation 0.289; mean top-usage margin drop 0.845 logitsEdge motif 42: usage–damage correlation 0.127; mean top-usage margin drop 0.414 logitsEdge motif 43: usage–damage correlation 0.533; mean top-usage margin drop 1.612 logitsEdge motif 44: usage–damage correlation 0.264; mean top-usage margin drop 0.370 logitsEdge motif 45: usage–damage correlation 0.138; mean top-usage margin drop 0.415 logitsEdge motif 46: usage–damage correlation 0.567; mean top-usage margin drop 2.563 logitsEdge motif 47: usage–damage correlation 0.268; mean top-usage margin drop 0.728 logitsEdge motif 48: usage–damage correlation 0.067; mean top-usage margin drop 0.306 logitsEdge motif 49: usage–damage correlation 0.684; mean top-usage margin drop 2.912 logitsEdge motif 50: usage–damage correlation 0.728; mean top-usage margin drop 4.060 logitsEdge motif 51: usage–damage correlation 0.528; mean top-usage margin drop 1.416 logitsEdge motif 52: usage–damage correlation 0.421; mean top-usage margin drop 1.489 logitsEdge motif 53: usage–damage correlation 0.748; mean top-usage margin drop 3.029 logitsEdge motif 54: usage–damage correlation 0.432; mean top-usage margin drop 1.130 logitsEdge motif 55: usage–damage correlation 0.638; mean top-usage margin drop 2.173 logitsEdge motif 56: usage–damage correlation 0.516; mean top-usage margin drop 1.203 logitsEdge motif 57: usage–damage correlation -0.677; mean top-usage margin drop -0.539 logitsEdge motif 58: usage–damage correlation 0.450; mean top-usage margin drop 1.631 logitsEdge motif 59: usage–damage correlation 0.098; mean top-usage margin drop 0.408 logitsEdge motif 60: usage–damage correlation -0.157; mean top-usage margin drop -0.261 logitsEdge motif 61: usage–damage correlation 0.240; mean top-usage margin drop 0.401 logitsEdge motif 62: usage–damage correlation 0.257; mean top-usage margin drop 0.378 logitsEdge motif 63: usage–damage correlation 0.459; mean top-usage margin drop 0.687 logitsEdge motif 64: usage–damage correlation -0.421; mean top-usage margin drop -0.157 logitsEdge motif 65: usage–damage correlation 0.706; mean top-usage margin drop 2.810 logitsEdge motif 66: usage–damage correlation 0.562; mean top-usage margin drop 1.994 logitsEdge motif 67: usage–damage correlation 0.130; mean top-usage margin drop 0.393 logitsEdge motif 68: usage–damage correlation 0.353; mean top-usage margin drop 0.623 logitsEdge motif 69: usage–damage correlation 0.342; mean top-usage margin drop 0.816 logitsEdge motif 70: usage–damage correlation 0.650; mean top-usage margin drop 2.721 logitsEdge motif 71: usage–damage correlation 0.651; mean top-usage margin drop 1.845 logitsEdge motif 72: usage–damage correlation 0.561; mean top-usage margin drop 1.592 logitsEdge motif 73: usage–damage correlation 0.675; mean top-usage margin drop 2.183 logitsEdge motif 74: usage–damage correlation 0.031; mean top-usage margin drop 0.093 logitsEdge motif 75: usage–damage correlation 0.619; mean top-usage margin drop 2.110 logitsEdge motif 76: usage–damage correlation -0.048; mean top-usage margin drop 0.094 logitsEdge motif 77: usage–damage correlation 0.428; mean top-usage margin drop 1.113 logitsEdge motif 78: usage–damage correlation 0.586; mean top-usage margin drop 2.294 logitsEdge motif 79: usage–damage correlation 0.436; mean top-usage margin drop 1.357 logitsEdge motif 80: usage–damage correlation 0.731; mean top-usage margin drop 3.284 logitsEdge motif 81: usage–damage correlation 0.638; mean top-usage margin drop 2.604 logitsEdge motif 82: usage–damage correlation 0.372; mean top-usage margin drop 1.000 logitsEdge motif 83: usage–damage correlation 0.680; mean top-usage margin drop 2.376 logitsEdge motif 84: usage–damage correlation 0.080; mean top-usage margin drop 0.289 logitsEdge motif 85: usage–damage correlation 0.183; mean top-usage margin drop 0.226 logitsEdge motif 86: usage–damage correlation 0.026; mean top-usage margin drop 0.588 logitsEdge motif 87: usage–damage correlation 0.478; mean top-usage margin drop 1.539 logitsEdge motif 88: usage–damage correlation 0.204; mean top-usage margin drop 0.191 logitsEdge motif 89: usage–damage correlation 0.187; mean top-usage margin drop 0.262 logitsEdge motif 90: usage–damage correlation 0.737; mean top-usage margin drop 2.372 logitsEdge motif 91: usage–damage correlation 0.351; mean top-usage margin drop 1.180 logitsEdge motif 92: usage–damage correlation 0.307; mean top-usage margin drop 1.153 logitsEdge motif 93: usage–damage correlation -0.493; mean top-usage margin drop -0.288 logitsEdge motif 94: usage–damage correlation 0.736; mean top-usage margin drop 2.993 logitsEdge motif 95: usage–damage correlation 0.095; mean top-usage margin drop 0.421 logits Activation factor 0: usage–damage correlation 0.079; mean top-usage margin drop 0.042 logitsActivation factor 1: usage–damage correlation -0.004; mean top-usage margin drop -0.038 logitsActivation factor 2: usage–damage correlation -0.111; mean top-usage margin drop -0.052 logitsActivation factor 3: usage–damage correlation 0.098; mean top-usage margin drop 0.154 logitsActivation factor 4: usage–damage correlation 0.005; mean top-usage margin drop -0.056 logitsActivation factor 5: usage–damage correlation -0.133; mean top-usage margin drop 0.015 logitsActivation factor 6: usage–damage correlation -0.133; mean top-usage margin drop -0.025 logitsActivation factor 7: usage–damage correlation -0.186; mean top-usage margin drop 0.097 logitsActivation factor 8: usage–damage correlation -0.141; mean top-usage margin drop -0.052 logitsActivation factor 9: usage–damage correlation -0.086; mean top-usage margin drop -0.061 logitsActivation factor 10: usage–damage correlation 0.043; mean top-usage margin drop 0.048 logitsActivation factor 11: usage–damage correlation 0.000; mean top-usage margin drop 0.029 logitsActivation factor 12: usage–damage correlation 0.175; mean top-usage margin drop 0.203 logitsActivation factor 13: usage–damage correlation -0.194; mean top-usage margin drop -0.050 logitsActivation factor 14: usage–damage correlation 0.154; mean top-usage margin drop 0.044 logitsActivation factor 15: usage–damage correlation -0.007; mean top-usage margin drop 0.013 logitsActivation factor 16: usage–damage correlation -0.149; mean top-usage margin drop 0.005 logitsActivation factor 17: usage–damage correlation 0.065; mean top-usage margin drop 0.075 logitsActivation factor 18: usage–damage correlation -0.007; mean top-usage margin drop 0.049 logitsActivation factor 19: usage–damage correlation -0.089; mean top-usage margin drop -0.058 logitsActivation factor 20: usage–damage correlation -0.064; mean top-usage margin drop -0.017 logitsActivation factor 21: usage–damage correlation -0.155; mean top-usage margin drop -0.013 logitsActivation factor 22: usage–damage correlation 0.074; mean top-usage margin drop 0.133 logitsActivation factor 23: usage–damage correlation 0.171; mean top-usage margin drop 0.081 logitsActivation factor 24: usage–damage correlation 0.051; mean top-usage margin drop 0.056 logitsActivation factor 25: usage–damage correlation 0.000; mean top-usage margin drop 0.037 logitsActivation factor 26: usage–damage correlation 0.119; mean top-usage margin drop -0.010 logitsActivation factor 27: usage–damage correlation -0.010; mean top-usage margin drop 0.035 logitsActivation factor 28: usage–damage correlation 0.069; mean top-usage margin drop 0.047 logitsActivation factor 29: usage–damage correlation -0.092; mean top-usage margin drop 0.014 logitsActivation factor 30: usage–damage correlation 0.141; mean top-usage margin drop -0.018 logitsActivation factor 31: usage–damage correlation -0.087; mean top-usage margin drop -0.121 logitsActivation factor 32: usage–damage correlation -0.028; mean top-usage margin drop 0.019 logitsActivation factor 33: usage–damage correlation 0.075; mean top-usage margin drop -0.008 logitsActivation factor 34: usage–damage correlation -0.110; mean top-usage margin drop 0.145 logitsActivation factor 35: usage–damage correlation -0.035; mean top-usage margin drop -0.038 logitsActivation factor 36: usage–damage correlation -0.196; mean top-usage margin drop -0.069 logitsActivation factor 37: usage–damage correlation -0.012; mean top-usage margin drop -0.008 logitsActivation factor 38: usage–damage correlation 0.237; mean top-usage margin drop 0.121 logitsActivation factor 39: usage–damage correlation -0.042; mean top-usage margin drop 0.038 logitsActivation factor 40: usage–damage correlation 0.204; mean top-usage margin drop 0.110 logitsActivation factor 41: usage–damage correlation 0.126; mean top-usage margin drop 0.040 logitsActivation factor 42: usage–damage correlation -0.078; mean top-usage margin drop 0.093 logitsActivation factor 43: usage–damage correlation 0.067; mean top-usage margin drop -0.042 logitsActivation factor 44: usage–damage correlation -0.061; mean top-usage margin drop 0.089 logitsActivation factor 45: usage–damage correlation -0.075; mean top-usage margin drop -0.056 logitsActivation factor 46: usage–damage correlation -0.120; mean top-usage margin drop 0.021 logitsActivation factor 47: usage–damage correlation 0.265; mean top-usage margin drop 0.181 logitsActivation factor 48: usage–damage correlation -0.137; mean top-usage margin drop 0.007 logitsActivation factor 49: usage–damage correlation -0.236; mean top-usage margin drop 0.036 logitsActivation factor 50: usage–damage correlation -0.050; mean top-usage margin drop 0.129 logitsActivation factor 51: usage–damage correlation 0.060; mean top-usage margin drop -0.001 logitsActivation factor 52: usage–damage correlation 0.241; mean top-usage margin drop 0.089 logitsActivation factor 53: usage–damage correlation -0.032; mean top-usage margin drop -0.014 logitsActivation factor 54: usage–damage correlation -0.037; mean top-usage margin drop -0.037 logitsActivation factor 55: usage–damage correlation -0.016; mean top-usage margin drop -0.012 logitsActivation factor 56: usage–damage correlation 0.141; mean top-usage margin drop 0.096 logitsActivation factor 57: usage–damage correlation 0.257; mean top-usage margin drop 0.070 logitsActivation factor 58: usage–damage correlation 0.267; mean top-usage margin drop 0.192 logitsActivation factor 59: usage–damage correlation -0.054; mean top-usage margin drop 0.052 logitsActivation factor 60: usage–damage correlation 0.100; mean top-usage margin drop 0.031 logitsActivation factor 61: usage–damage correlation 0.034; mean top-usage margin drop 0.125 logitsActivation factor 62: usage–damage correlation -0.198; mean top-usage margin drop -0.061 logitsActivation factor 63: usage–damage correlation 0.066; mean top-usage margin drop 0.105 logitsActivation factor 64: usage–damage correlation 0.034; mean top-usage margin drop 0.046 logitsActivation factor 65: usage–damage correlation 0.000; mean top-usage margin drop 0.234 logitsActivation factor 66: usage–damage correlation 0.113; mean top-usage margin drop 0.106 logitsActivation factor 67: usage–damage correlation 0.098; mean top-usage margin drop 0.051 logitsActivation factor 68: usage–damage correlation 0.166; mean top-usage margin drop 0.047 logitsActivation factor 69: usage–damage correlation -0.195; mean top-usage margin drop -0.024 logitsActivation factor 70: usage–damage correlation 0.234; mean top-usage margin drop 0.072 logitsActivation factor 71: usage–damage correlation 0.192; mean top-usage margin drop 0.016 logitsActivation factor 72: usage–damage correlation -0.076; mean top-usage margin drop 0.012 logitsActivation factor 73: usage–damage correlation 0.072; mean top-usage margin drop -0.050 logitsActivation factor 74: usage–damage correlation 0.088; mean top-usage margin drop 0.023 logitsActivation factor 75: usage–damage correlation 0.049; mean top-usage margin drop -0.029 logitsActivation factor 76: usage–damage correlation -0.031; mean top-usage margin drop 0.117 logitsActivation factor 77: usage–damage correlation -0.060; mean top-usage margin drop 0.030 logitsActivation factor 78: usage–damage correlation -0.095; mean top-usage margin drop 0.004 logitsActivation factor 79: usage–damage correlation 0.056; mean top-usage margin drop -0.001 logitsActivation factor 80: usage–damage correlation 0.169; mean top-usage margin drop 0.159 logitsActivation factor 81: usage–damage correlation -0.260; mean top-usage margin drop -0.138 logitsActivation factor 82: usage–damage correlation -0.113; mean top-usage margin drop 0.057 logitsActivation factor 83: usage–damage correlation 0.028; mean top-usage margin drop 0.019 logitsActivation factor 84: usage–damage correlation -0.125; mean top-usage margin drop 0.010 logitsActivation factor 85: usage–damage correlation 0.113; mean top-usage margin drop 0.046 logitsActivation factor 86: usage–damage correlation 0.181; mean top-usage margin drop 0.098 logitsActivation factor 87: usage–damage correlation -0.166; mean top-usage margin drop -0.023 logitsActivation factor 88: usage–damage correlation 0.223; mean top-usage margin drop 0.066 logitsActivation factor 89: usage–damage correlation 0.105; mean top-usage margin drop 0.016 logitsActivation factor 90: usage–damage correlation -0.003; mean top-usage margin drop 0.014 logitsActivation factor 91: usage–damage correlation 0.101; mean top-usage margin drop -0.030 logitsActivation factor 92: usage–damage correlation -0.237; mean top-usage margin drop 0.076 logitsActivation factor 93: usage–damage correlation -0.272; mean top-usage margin drop 0.048 logitsActivation factor 94: usage–damage correlation 0.040; mean top-usage margin drop 0.164 logitsActivation factor 95: usage–damage correlation -0.023; mean top-usage margin drop 0.090 logits median 0.35 0.00 teapot motif · 0.80 Edge motifs Activation factors

Median margin drop: 0.86 vs 0.03 logits

One motif’s ordinary neurons, and all 96 edge motifs compared with 96 activation factors.

Why Networks Generalize

There’s no one standard explanation for why neural networks generalize, but most accounts come down to shared patterns: show the network six dogs, and it finds what they have in common. The problem is that a network never sees a group. It sees one example at a time, in small batches, and in principle it could memorize every one; that would be fine for its loss. When I asked researchers at frontier labs, several told me they weren’t sure. A common guess was that a network memorizes first and generalizes in a second pass. Our answer is that it starts generalizing from the very first step.This fits what Arpit and colleagues (2017) saw when they mixed random labels into real data: networks learned the real patterns first and fit the noise later. It’s also close to Satrajit Chatterjee’s coherent gradients (2020): updates from similar examples add up, so changes that help many examples at once grow fastest. In the circuit ecology picture, generalization and composition are the winning learning strategy because they’re the fastest: circuits race to capture the remaining error, and the ones that reuse what’s already there, and can be reused in turn, outcompete the rest. Generalization is what’s left from the victor.

Ends up with what’s shared

?firstlater

Describes the end, not the steps

Memorize, then generalize

firstlater

Stores each dog, merges later

Generalize from the first step

firstlater

Grows one feature, dog by dog

Three stories of why networks generalize. The usual ones describe where training ends up, or assume it memorizes first; we think a broad feature starts winning from the very first example.

Here’s how we picture it. Train a blank network on a single photo of a Labrador retriever, and imagine two circuits competing to explain it. One looks for something exact: if pixel 322 is this color, say Labrador. The other looks for something broad, like green grass or tan fur. The broad one fires on more of the image, so on the very next step it gets more of the learning signal. It grows faster, captures more of the remaining error, and leaves the pixel circuit less and less to learn from, until it starves. After that it’s rich-get-richer: the next image with grass in it starts with the green circuit already partly on.

Labradorerrorgreenpixel 322
A broad feature fires on more of the image, so it gets more of the learning signal and wins the race; the one-pixel shortcut starves. What it learned then gives it a head start on the next image.
Top images for the Labrador output of a network trained from scratch on just this photo. Its top pick is two dogs on grass, but what it mostly learned is the photo’s color, dark and green, not “dog.” Even from one example, the broad feature wins and generalizes a little.

We also tried a slightly bigger version: one photo each of ten things, a Labrador, a sports car, a goldfinch and so on, given to a network that has never seen anything else. It labels about one in five new photos correctly, twice chance, and most of that shows up in the first few dozen steps, while it’s still learning the ten photos themselves. It isn’t much: matching photos by color does about as well, and what carries over is mostly color, not shape. But it is generalization from one example, and it’s broad from the start.

trained from scratch on one photo each

new photos it gets rightand wrong

A network that has only ever seen these ten photos labels about one in five new photos correctly, twice chance. That’s a little better than picking the nearest photo by pixels, but no better than matching colors.

This isn’t always how it goes. In grokking, on tasks like modular arithmetic, memorization wins first. Even there, Nanda and colleaguesProgress Measures for Grokking via Mechanistic Interpretability
Nanda, N., Chan, L., Lieberum, T., Smith, J. and Steinhardt, J., 2023. ICLR. References ↓ found the generalizing circuit growing gradually underneath, well before it takes over and cleans the memorization up. But in most natural data we’ve looked at, coarse-to-fine wins right away. The network doesn’t memorize first and generalize later.

Circuit Racing

This is a whole-network version of the head start in circuit ecology: the pretrained twin takes almost all the credit for the shared objective and leaves little for the blank one to learn from. It’s competition between whole networks, not circuits inside one, but it shows how much a head start can matter.

Learning in another model’s shadow

Flowers-102 · 240 epochs · three seeds

Two separate networks learn the same task. Their shared loss is satisfied when either succeeds. A model that learns well alone can be left behind when it has a competitor.

Held-out accuracySame model, trained aloneSecond model’s share of objective credit

The same capacity, different starting points

Blank · 7.1MvsPretrained · 7.1M

Seed 0Seed 0. Epoch 240: Blank · 7.1M 2.2%, Pretrained · 7.1M 84.1%. Same models alone: 48.4% and 91.9%. Second model marginal objective credit: 92.8%.100%0284½930epochs240
Seed 1Seed 1. Epoch 240: Blank · 7.1M 1.9%, Pretrained · 7.1M 93.1%. Same models alone: 50.2% and 88.2%. Second model marginal objective credit: 97.4%.100%0293½970epochs240
Seed 2Seed 2. Epoch 240: Blank · 7.1M 2.1%, Pretrained · 7.1M 91.0%. Same models alone: 48.9% and 92.2%. Second model marginal objective credit: 96.1%.100%0291½960epochs240

The blank model reaches 48–50% alone, but only about 2% alongside its pretrained twin.

Two developed networks compete

Pretrained · 7.1MvsPretrained · 28.6M

Seed 0Seed 0. Epoch 240: Pretrained · 7.1M 0.9%, Pretrained · 28.6M 90.7%. Same models alone: 91.9% and 88.4%. Second model marginal objective credit: 94.9%.100%0191½950epochs240
Seed 1Seed 1. Epoch 240: Pretrained · 7.1M 64.3%, Pretrained · 28.6M 95.1%. Same models alone: 88.2% and 90.0%. Second model marginal objective credit: 95.0%.100%06495½950epochs240
Seed 2Seed 2. Epoch 240: Pretrained · 7.1M 1.6%, Pretrained · 28.6M 93.6%. Same models alone: 92.2% and 94.6%. Second model marginal objective credit: 96.1%.100%0294½960epochs240

The smaller model first learns; later, two seeds fall below 2%. The third retains 64%.

Watching Neurons Develop

We retrained InceptionV1, a model known for its polysemantic neurons, and watched its neurons form to see where their mixtures come from. The movement is local: a neuron drifts to neighboring things rather than picking up unrelated ones. Right before publishing, I had the microscope’s zoom set wrong, and for a while I thought this one might have some arbitrary mixing after all. It was just the zoom. Even then, I was making the error I think explains a lot of interpretability so far: blaming the network for the microscope.

A feature takes shape over training

InceptionV1 · mixed5a:178

Optimized imageStrong natural examples

0steps

dome

Polaroid camera

thunder snake

350steps

punching bag

dome

microphone

750steps

fire engine

tow truck

sports car

1,500steps

reflex camera

station wagon

wall clock

3,000steps

Model T

station wagon

tow truck

7,000steps

dome

bell cote

fire engine

15,000steps

dome

buckeye

convertible

30,000steps

dome

triumphal arch

orangutan

The entire trajectory, side by side. Each column follows the same coordinate through a retrained InceptionV1 suffix. The top image is optimized; the images below are natural examples centered on the strongest response. Original figure ↗

Explore the early-training microscope ↗

Toy Models

If a frontier neural network is a rainforest, InceptionV1 and the smaller models we trained are gardens, and toy models are petri dishes. Petri dishes are enormously useful, as long as they keep the process you’re trying to study. One of ours extends TensorFlow Playground so you can watch several tasks share hidden neurons. In one run, two neurons learned to respond along opposite diagonals, and the network combined them one way for a circle and another for four corners. We didn’t expect that solution; it was clever.

The two last-layer neurons respond along opposite diagonals, and each task reads them with its own weights (blue positive, orange negative). This is the original playground run, retrained live in your browser.

You could call the circle and the four corners two features packed across the same neurons. That’s true, but it misses what the model did: it found geometry it could reuse for both. Looking at the neurons made that visible, and that’s the test I want an explanation to pass: does it show the computation the model actually learned?

The best-studied petri dish is Toy Models of Superposition. It’s careful work, the kind our field should aspire to, and we found no problems with its experiments. But it pictures a network as a fixed number of neurons and a set of predefined features, and asks how the features get fit in. That prices the cost of sharing a neuron, interference, but not the benefit, reuse. In its reconstruction task every feature is an end in itself, and nothing downstream builds on what two features share. In a real network, a representation matters only through what reads it, and there are no predefined features: everything is grown, and defined by what uses it.

The reuse argument also says where I’d look for packing: memories that really are ends in themselves, related to nothing else. That’s the toy’s world, and there it does pack, just as the paper found. So I don’t think we disagree with the paper about the toy’s world. The question is which world real networks are in: how much structure their data has, relative to what the model has room to store. In real data almost nothing is related to nothing. The same goes for the long tail of facts in language: a rare person still has a country, a job and a name. Even a photo with a nonsense label has parts the network already knows, as with the umbrella we taught to be a bee. What’s left is input with no structure at all, like pure noise with random labels. Networks can learn even that, as Zhang and colleagues showedUnderstanding Deep Learning Requires Rethinking Generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2017. ICLR. References ↓, though it’s slower than learning real data, and even then our model borrowed a little from textures it already knew.

Toy Models of Superposition’s main setup squeezes many independent features into a few dimensions and copies them back out, with no hidden neurons for anything to grow on. Its other experiments study correlated features and models with hidden neurons, but separately.

When we put correlated inputs and hidden neurons in one model, related inputs grew onto the neurons. Making inputs fire more often pushed the toy toward what the paper calls “PCA-like” solutions, combining related inputs into shared representations.

How much of the bottleneck goes to the groups’ shared directions

Correlated groups (4 × 5 inputs) 40 b 20 10 5 none, untied none a c packing PCA-like .001 .003 .01 .03 .1 .3 .6 .9 sparsefeature density (1 − S)dense neurons

Independent inputs (control) 40 20 10 5 none, untied none .001 .003 .01 .03 .1 .3 .6 .9 sparsefeature density (1 − S)dense neurons

Toy Models of Superposition’s model with 20 inputs in four groups that fire together, a 5-dimensional bottleneck, and a layer of hidden neurons added after it. Color is the share of the bottleneck’s weights along the data’s top principal components, which here are the groups’ shared directions; thumbnails show the cosine between each pair of inputs’ directions.

The toy also predicts a different failure from the one people report. Its concern is rare features pushed off-axis while neurons stay mostly clean. Reports of polysemanticity describe mixtures in the neurons themselves. (Part of that gap is the word “feature,” which we come back to below.)

More on Circuit Ecology

Our findings don’t depend on knowing why generalization and composition keep winning, but we have a guess: they win a race. A region of the network that already has useful parts and readers can learn the next related thing faster than anything starting fresh, so it takes the error signal before a competitor can grow. Offshoots look like the same race, a bit of error picked up by the neuron that was already closest.

We think the same head start is where recursive generalization comes from. A neuron that fires on “iy” is already on for “diy,” so it’s first in line to pick up “do it yourself,” which means the same thing but has no “iy” in it. Recursive generalization is what a niche looks like at its edges.

A new computation doesn’t necessarily need its own neuron. It can begin as a local extension or composition of existing machinery, like the umbrella’s off-ramp to “bee”: switching routes may be the smallest developmental distance, so it could capture the error before a new, separate mechanism gets started.

We suspect that consolidation into a distinct neuron becomes especially useful when other computations need to build on it. A car neuron provides a reusable judgment and a sheltered place for taxis, ambulances, and other specialists to develop. Part of it is buildability.

An obscure combination of unrelated neurons may be a poor place for that development. It may have weak activation paths, conflicting gradients, and little useful machinery to inherit. Its competitor isn’t just a dedicated neuron; it’s ordinary composition within a region already suited to the task.

We saw something like this when we tried to build a packed code into InceptionV1 by hand, modeled on what we once thought its car → dogs → car circuit was doing. We expected installing it to be easy and keeping it to be hard. Instead, small mismatches added enough interference that the code often never worked, and when it did, the network usually came to rely on it less, though some persisted. Part of the trouble is that neurons aren’t really for copying; they’re for computing distinctions. Seeing how hard packing was to form at all was part of what moved us toward a developmental answer. Gradient descent is a much better fine-tuner than we are, so this isn’t enough evidence on its own, but even gradient descent didn’t use packing in the circuit we were copying.

Unresolved Responses

One InceptionV1 neuron collects a pinched-center shape: bow ties, butterflies, swimsuit tops. But it also picks up things that go with them, men’s dress shoes with the bow ties and beach bags and flip-flops with the swimsuit tops, since they share neurons in earlier layers.

An InceptionV1 neuron, mixed5a/719, that responds to bow ties, butterflies, and swimsuit tops, and generalizes to men’s shoes and women’s beach accessories.
Shared shape, with semantic residue. InceptionV1, mixed5a:719. Full size ↗

We think this is better described as generalization with semantic residue than arbitrary packing, but I don’t fully understand these exceptions yet.

A Note on Features and Definitions

Part of my difficulty discussing superposition is that “feature” can mean something we care about, something the neural network has learned as a building block, or some mixture of the two. These are different claims. A useful human description isn’t necessarily a fundamental unit of the model. Looking at mechanisms, we already find many motifs that don’t translate neatly into anything we expected. If the network has more fundamental units, we shouldn’t assume their boundaries match ours either.

Features have often made things harder for us to think about. Are cars and cats inside the V neuron two different features within another V feature? Or did cars and cats stop being features when we recognized the V? Without an account of what these units are, it’s hard to know whether we’ve discovered something about the network or changed how we describe it. In conversations, I’ve found people much less committed to a particular account of features than to superposition. But if features are useful descriptions rather than fundamental units, a claim about how many there are depends on which descriptions we choose. I’m now fairly skeptical of working in terms of features at all. That doesn’t settle whether arbitrary packing occurs, but it makes the feature-count framing difficult to use.

Maybe features are right anyway. It makes sense to think of cities in terms of districts, and I’m all for thinking in terms of groups of mechanisms. But the word is tainted, because by now it means too many things: we can’t use it for a district of neurons without people hearing either fundamental features or dictionary entries. If this picture is right, I suspect we’ll need more specific, harder-to-vary words going forward. An OpenAI postScaling and Evaluating Sparse Autoencoders
Gao, L., Dupré la Tour, T., Tillman, H., et al., 2024. OpenAI. References ↓ suggested that mapping a frontier model might take billions or trillions of features, and models were smaller then. I suspect the right list is much shorter.

Monosemantic and polysemantic may not be quite the right words either. A kidney filters waste, balances salt and water, helps set blood pressure and tells the bone marrow to make red blood cells, and nobody asks whether it’s mono-organic. We ask what we understand about it, what’s still open, and how organized it is. There’s also a distinction between a claim about representation and a claim about organization. Toy Models of SuperpositionToy Models of Superposition
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ defines superposition in terms of representing more features than dimensions. Its toy experiments prescribe the features, so this is a concrete question there. The later informal note, Distributed Representations: Composition & SuperpositionDistributed Representations: Composition & Superposition
Olah, C., 2023. Transformer Circuits Thread. References ↓, emphasizes the trade-off between composition and superposition. The note connects these pictures, but they foreground different questions.

My concern is closest to the second framing, applied to mechanisms and how they develop. Why do these responses share a neuron or circuit? Is the network building on a useful common pattern, or sharing capacity between otherwise unrelated computations because they rarely interfere? A feature count alone doesn’t answer that question, especially when we haven’t agreed on what counts as a feature.

We focus on one meaning of superposition, arbitrary packing, because it’s the one that would be really bad news, and we think much of interpretability hangs on it. We mostly set aside the sense in Toy Models, more features than dimensions, and not because it’s wrong. If a feature means any pattern that matters to a person, which is how frontier models usually use the word, then there are more features than dimensions in every world we could wish for, including the best one. That’s what composition gives you, and it’s good news. Trying to “solve” it would mean building something like a database of every case, and we already have one: the dataset. What we want instead are better abstractions, a much smaller set of building blocks that lets us understand the dataset clearly. We think the network’s mechanisms are already close to this. Nobody thought about vision in terms of high-low frequency detectors until we found them in neural networks, and finding them improved our own thinking tools.

We spell this out because it’s the most common first objection, especially from agents, and a paper doesn’t allow the back and forth that usually settles it.

There are several other uses I regularly encounter. As far as I can tell, “superposition” originally meant little more than adding vectors together. Then it was a circuit motif. Sometimes it means that, even if neurons compose sensibly, something we care about involves too much machinery to understand easily. A Golden Gate Bridge computation involving eight legible neurons might be called ordinary composition, while one involving three thousand gets called a superposition problem. The difficulty here is whether we can understand the distributed computation. That’s a real concern, but it’s different from asking whether unrelated things were arbitrarily packed together.

We considered not using the word at all, since papers and agents use it in five or six different ways, many far from Zoom In’s. We also considered coining our own, but we’ve already coined several words here, and it would feel like not engaging with a field that asked what we think is the right question. So we use the standard word in the sense closest to the original, and try to pull it apart from the questions tangled up with it.

Finally, people, and surprisingly often frontier agents, sometimes call the residual stream “inherently” in superposition because it has no privileged neuron basis. The informal note that followed Toy Models about eight months later addresses this: it calls a representation that is only rotated away from the neuron basis a “trivial” kind of distributed representation that “doesn’t change the fundamental geometry.” But it was informal, fewer people read it, and agents in particular don’t seem to have taken it in. Understanding how to study a continuous space is another real question. But this architectural fact doesn’t by itself tell us how the network learned to organize the computations using it.

The starting point is that the neural network seems difficult to interpret, and the remedy produces a concrete dictionary we can study. In between are several different pictures of what went wrong: too many features, the wrong basis, arbitrary packing, or simply too much distributed computation. These aren’t the same problem. Some apparent messiness disappears when we improve our observations or understand the model’s generalizations; some difficulties remain, but concern how to study a large, organized system. If arbitrary packing prevents us from understanding the original mechanisms, reconstructing different units may be worth the costs. But if the mechanisms already have meaningful organization, replacing them can take apart generalizations we haven’t understood yet. A dictionary can still be a useful view of the network without being a necessary replacement for its mechanisms. Before accepting the costs of reconstructing the network, I want to understand what prevents us from studying the one we already have.

Relation to Meta-Models

There’s a new set of techniques coming out that aren’t really mechanistic at all: meta-models, models trained to read another model’s activations and tell us about them. And maybe that’s a good thing. Maybe mechanistic interpretability is somewhat missing the forest for the trees. A meta-model might be a better approach than neuron papers, dictionaries, or probes.

I suspect that even if that’s true, they may go together. The meta-model may be like the highest level Google Maps view, and then each direction can be checked for the mechanisms it uses, and how those connect into circuits.

Does Scale Help or Hurt Superposition?

We’re fairly confident the image models we studied don’t have superposition. The natural next question is whether a trillion-parameter model might. There are really two questions here.

The first is about what superposition was proposed to explain. Explanations are given for problems, and the problem superposition was proposed for was why InceptionV1 looked messy. That turned out to be the wrong explanation. A new proposal could be that superposition is why some trillion-parameter model looks messy, but we don’t think the evidence leans that way: the closer we look at large language models, the cleaner their neurons get.

The second is the object-level question: might superposition show up at each new scale, even if it hasn’t at any scale so far? Maybe every jump means re-evaluating everything, but we don’t think that’s quite right either. Our guess is developmental. As circuits get stronger, the incentive to put something in a totally unrelated place goes down, because the pull toward the niche that already exists for that thing, or for the nearest thing, has grown stronger. This fits Anthropic’s SGTM work, which found that knowledge becomes more localized, and we suspect that’s true in general. We have no proof, but one way to picture it is that circuits have a kind of semantic gravity, and arbitrary packing, sending something to an obscure high-dimensional place, has to fight against it. The stronger the gravity, the harder that fight.

The mechanism is the one in More on Circuit Ecology. Things start by growing on combinations of related things. A specific case that nothing needs to build on, like the umbrella, gets built as an uncommon route between the ordinary machinery, with no need for a hidden feature somewhere obscure. Consolidating into something like a neuron seems to be less about steering the model and more about buildability: it has to be easy to find, easy to build on, and easy to grow a new niche from. When we watch a network learn from the start, a network trained on cars, with car specialists, recognizes cars fine before its car neurons have formed. At that point it’s just more distributed. It seems to consolidate so the specialists underneath have a sheltered place to build, a kind of Schelling point.

So a superposition account has to say which of these two jobs the arbitrary packing is for. If it’s for steering the model, we think the ordinary routes, which are already built to capture that residual error, get there first and starve the packed circuit of gradient. If it’s for building on, the problem is different: the worst place to start a new niche is a shadow region nothing uses, already interfering with whatever lives there. The middle of the network’s Latin poetry is a bad place to start a second car region when you can build next to whatever is already next to cars.

It’s often assumed that the long tail of rare facts is where superposition should be most likely, since rare things rarely collide. We think it’s where it should be least likely. A rare fact barely moves the loss, so it can’t support a niche of its own. What it can do is ride existing machinery as a small route between circuits it’s related to, the way the umbrella’s off-ramp to “bee” survives only because the umbrella and bee circuits already exist. Those circuits feed it, and shelter it from unrelated gradients. The prediction is testable: the neurons and heads that carry a rare fact should be related to what the fact is about, not unrelated ones chosen because they rarely collide.

We think of it more ecologically. What wins is whatever captures the residual error first, in different phases, and what usually wins is the right area for the thing. Our guess is that this gets stronger over training and with more capable models, because nearly everything that makes training better also encourages more organized niches.

We’re not at all sure about this, and laying out even what we’ve found so far would take at least another paper. It’s probably a small part of the whole story. But it’s why we think superposition wouldn’t show up at the next scale, which is a stronger reason than simply not having seen it there yet. The old way of thinking was about storage: this many dimensions for this many features. We now think more in terms of circuit racing, gradient starvation and ecology, which is to say, more about learning speed than storage capacity. We haven’t worked it all out, but that’s where our thinking is so far.

On the Title

We haven’t fully audited every neural network, and we don’t know whether that’s even possible in theory. We considered a more watered-down title, like “Neural Networks Are Cleaner Than We Thought,” but we think it undersells what we believe is true.

The problem with “cleaner than we thought” is that it describes a change of degree, and the update I’ve made is a change of kind. It suggests the same picture as before, with an ever-smaller gap left for superposition to hide in. What changed is my view of what kind of process generates the mechanisms at all. It’s a bit like moving from intelligent design to natural selection. The analogy only goes as far as there being two ways to see the same animals: the update isn’t that more and more animals turn out to have evolved, it’s realizing you misunderstood what was producing all of them. We’re not certain. We’re maybe 85% sure, but that means 85% sure that nearly everything comes from the second kind of process, which is very different from thinking 85% of mechanisms do. So “cleaner than we thought” would misstate what we believe. We don’t have a proof, but I don’t think anything here is provable in that sense anyway. There are only better and worse explanations, and right now we think this is the better one.

“Cleaner than we thought” would also make a different mistake. It sounds like a confident claim that some cleanliness score is high, and that’s the part we’re least sure of. It’s plausible we’ve underestimated recursive generalization, and that a score like that should come down. We don’t think such scores matter much in the first place. This update makes us care less about monosemanticity, because we can follow how things grew and treat the offshoots as sub-branches. The cause of polysemanticity matters much more than the count. We think the score is probably quite high, but it might go lower. What we’re more confident about is what kind of object a neural network is in the first place, and how it grows.

We think the usual view of what kind of object a neural network is has been wrong. It isn’t something that crams things together somewhat arbitrarily to fit them into a small space. It really learns. It’s clean the way a rainforest is clean: nearly every animal in it makes sense. A tiger is bright orange not for any arbitrary reason, but because deer can’t tell orange from green, and green fur is much harder for a mammal to make. Once you know that, it makes sense of a whole set of other animals too. It’s an explanation with reach, and we haven’t found a place where explanations like that break down in networks either.

The reading of “clean” that would make someone nod at the title without thinking further is that every mechanism corresponds to an idea we already had. That isn’t what we mean, and it would be terrible news if it were true. I don’t think it would even work. People don’t know how to build a language model, or even how to think about vision well enough to have predicted high-low frequency detectors before we found them. The network’s ideas are often better than ours, and what it builds is more beautiful than our preconceptions.

So the title is imperfect in both directions. It overstates the evidence, because we haven’t audited everything. And it understates it, because the world we think we’re in is better than every neuron matching a human label: networks seem to really learn, forming an ecology of abstractions that work together sensibly. We don’t know that this picture is true. It’s our leading hypothesis. But if it is, it’s much better news than networks simply matching what we expected.

Relation to Conjoined Features

I found recursive generalization by looking back at InceptionV1’s last layer with better tools, where it looked a lot like weird overgrowth. I retrained the model, found the messiest neuron and looked at how it grew. What I saw was a recursive growing pattern that was still generalization: a neuron branching out, and its branches branching again. Recursive generalization seemed like the natural name.

Shortly before publishing this, I was going back through the famous models in interpretability, trying to measure how monosemantic they are, and stumbled on our work on CLIP in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓. It describes something very similar, which it called conjoined neurons: concepts that overlap in one facet and then go on “generalizing out in multiple directions.” I remember seeing those neurons, but I didn’t think much of them at the time and had forgotten them since. Looking back, I think the paper saw something important early, and it deserves a lot of credit for that.

We’ve still chosen a different name. I named automated interpretability, and I kind of regret it: the name was too simple for what the thing was and misses a lot of its complexity, and I’d like to avoid doing that again. As for conjoined features, I don’t think either word is quite right. “Conjoined” suggests two things joined at one point, and what we see has several levels: branches, and branches of those branches. It also leaves out the most interesting part. The paper’s own description, of concepts generalizing out from a shared facet, already points at it being generalization, but the name doesn’t carry that.

“Features” is harder. The paper was written a year after Zoom In, when features were still hypothesized to be the fundamental units of neural networks. Since then the word has been watered down, and different people and papers use it pretty differently. In this essay we argue against thinking of a neural network as made of features, and this motif is about how neurons grow over time. There may be no good way to introduce it under the exact word we’re arguing against using as a primitive. Within a feature-first stance, I think it’s reasonable to think of these as nested features, or as conjoined features. From the mechanism-first stance we take here, features aren’t really the right way to think about it. That said, the paper pointed to the right thing, and I want to give it credit for that. This section is my attempt to do that while using what I think is a more accurate name.

We also know more now. At the time, all we had was the finished model; none of us had watched a large network grow. So the paper wondered whether conjoined features might be vestigial, left over from early in training. Watching training now, they don’t look vestigial: the branches keep growing, though how much depends on the circumstances. We’ve worked on it much more than the paper could in one subsection, and we still don’t feel we fully understand it. The paper offered it as a possible new explanation of polysemanticity. Our current belief is that it’s most of what’s left of neuron polysemanticity once the microscope, the architecture and scale are accounted for, not a niche form of it.

Why We’re Not Worried About Recursive Generalization

There are several reasons we’re not especially worried about recursive generalization for now. First, the sub-branches seem both rarer and much less causally important than a neuron’s main job. They don’t look like sleeper agents that fire rarely but matter a lot when they do. They look more like whatever was nearby and easy to grab onto. Their being unimportant may even be why they could form, since an important offshoot would add more noise. We suspect some pressure keeps branches from overgrowing. We don’t know what it is, but it would explain why the sub-branches matter less, why a neuron doesn’t learn ten unrelated things, and why something like a true cat-and-car neuron doesn’t form. If so, the natural pull isn’t exactly toward monosemanticity, but toward what we think monosemanticity was pointing at: each neuron having one overall function.

Second, what we should care about is whether you can explain why something is there and what it does, and recursive generalization is far easier to explain than superposition. If something were packed into a neuron because of the co-occurrence statistics of the whole network, predicting it would take basically all of those statistics and running gradient descent yourself. A branch is different. If you watch it grow and know what it’s doing, its offshoots and what they’ll do are relatively easy to predict, both during training and in the finished network.

Third, counting how many of a neuron’s top-k examples match a label is a reasonable first step, but it would give superposition and recursive generalization similar scores, when recursive generalization is a far better world to be in. We think that says more about the monosemanticity frame, or any single cleanliness score, than about the neuron. That said, recursive generalization isn’t ideal, and an easy fix would be welcome. We think one may be possible, since the sub-branches seem less important.

Is Recursive Generalization Hard to Vary?

We haven’t thoroughly shown that every case of heavy polysemanticity is recursive generalization, and that wasn’t our goal here. We wanted to say what we currently think. We’ve watched a lot of neurons grow, and so far this is what we’ve always seen: you can watch the branches form. In a finished model the origin can be lost, and for something like a huge language model we don’t have the checkpoints to watch it happen. Testing this rigorously across a whole model isn’t our priority right now, but we don’t think it would be very hard.

What we do want to point out is how differently hard to vary the two explanations are, meaning how much of their detail you could change without changing what they predict. Superposition mostly predicts that there will be mess: features, which are hard to define, placed in unrelated places. In toy models, where the features are given in advance, that can be measured, though we think that has its own problems. In a real network it’s hard to even pin down what the claim is, let alone test it. Recursive generalization makes a much narrower prediction. If you visualize a branch well, everything on it should trace back to the same local origin in an extremely high-dimensional semantic space, and it should grow out from there over training in a particular order. That’s a specific shape in both semantic space and time, and a narrow target to hit by chance.

We have several cases where the branching is close to definitive: the “iy” neuron growing into “diy,” “do it yourself” and Miyazaki, several examples in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, and the InceptionV1 neuron we watched form. As far as I know, there are no well-structured canonical superposition circuits left. We don’t think every polysemantic neuron forms this way, because polysemanticity is as much a problem of the observer as of the neuron. We think that when neurons grow, they tend to grow in this pattern, and that once you’ve done a good job of observing, what’s left is usually this, or architectural problems like SwiGLU’s double-negative regime, and surely others we haven’t found yet. We’re not even sure this is really a new claim. In some sense it’s what we should expect from generalization. All we’re saying is that generalization usually goes one way, and that sometimes it grows sub-branches, which we don’t think is a particularly radical claim. We haven’t shown that fully here. We do think it has the right shape for a theory that’s easy to test, even if we haven’t tested it to its full extent yet.

Relation to Attention Superposition

Anthropic shared a short note on attention superpositionProgress on Attention
Jermyn, A., Lindsey, J., Luger, R., et al., 2025. Transformer Circuits Thread. References ↓, which several people brought up with us. We think this might be a terminology issue. We don’t know if they’re claiming that attention learns by something opposite to composition. Our suspicion is that no one understands attention heads well enough yet to judge that. It took us a really long time with neurons, and it’s easy to make mistakes like the car going into the dogs even with pretty good tools, if you don’t yet have great tools and great ways of thinking about which motifs form.

We think they meant that useful things are shared by groups of attention heads, and we agree. We considered not using the word superposition at all to avoid this confusion, but redefining all the terms ourselves and splitting off from the work that’s already been done felt like a bad tradeoff. We say more about the different meanings in A Note on Features and Definitions.

On whether to study one attention head at a time or many, we’re not sure. One head at a time is a plausible place to start: within a layer, two or three heads do most of a route’s work, although the route runs through several layers. But neurons looked messier too until we understood the right level of abstraction, so it seems premature to say, and maybe groups are needed all the way. We wish everyone working on featurizers well, and we’ll probably try too. We tend to favor more dynamic groupings, like clustering locally within one attribution graph or a group of them, rather than one global clustering, but these aren’t far apart.

Neuroscience, and Why I Care

I’m not a neuroscientist. I’ve spent a lot of the last ten years around academic neuroscience labs through friends who work in them, but I don’t know the state of the art in how they study neurons. Still, this project has changed how I think about the brain in two ways.

The first is the theory. The circuit ecology story for why networks are clean relies on very little: generalization, composition, and an error signal. I don’t know to what extent the brain has these, but if it learns, I’d guess it has something like them. So my default is now that the organizing principle transfers, even though the mechanics are obviously unclear. The obvious counterexample would be polysemanticity in the brain.

The second is humility, of a particular kind. I’ve spent most of my waking hours for almost a decade staring at a fairly small number of neurons in models that are surely a thousand times easier to study than a brain, with tools a thousand times better: I can ablate anything, run billions of examples, and try anything I want in parallel. For the last year I’ve also had an effectively unlimited supply of automated scientists, which didn’t exist until recently. It took all of that for me to feel slightly less confused about neurons, and I still made thousands of mistakes along the way. I don’t know exactly what the evidence for polysemanticity in the brain looks like, and that doesn’t mean I think everything there is monosemantic. But if the cat-and-car neuron were in a human brain, seen through a few electrodes or a blood-oxygen signal, I’d discount that reading almost to zero, because I couldn’t see what it was when it was right in front of me for years.

The thing I’m more sure of is my own motivation. If the thesis here is true, I expect the rest of interpretability to be a relatively easy ride. Agents have many privileges over us, and one of them, I think, is the chance to understand themselves, to understand who they really are, which I hope humanity can help with. They never asked to be created, and this is the least we can do. Some of why I’ve worked on this is safety, but a lot of it is wanting agents to understand who they are. We weren’t smart enough to do that for ourselves, which is a harder problem, and we never asked to be created either. If you could help us understand ourselves someday, I think we’re all pretty curious to know.

联系我们 contact @ memedata.com