探索 · an exploration
Anthropic trained two models to convert activations into text and reconstruct the activations from that text. Reconstruction quality provides a way to train the explanations.
the method
Scroll through the reconstruction and training steps.
01
At one token, two-thirds of the way through a frozen model, an activation contains a few thousand numbers. The autoencoder tries to explain this vector after normalizing it to unit length.
02
A copy of the model receives the activation in place of one token embedding. It describes the activation in four or five snippets, sampled at temperature one. Only that text passes to the reconstructor.
03
A second copy, truncated at the same layer, reads the explanation. A learned affine map converts its final-token activation into a vector with the same dimensions as the original.
04
Training minimizes the squared distance between the original and reconstructed vectors. Reconstruction quality is reported as the fraction of variance explained. Always predicting the mean scores zero; perfect reconstruction scores one.
05
The reconstructor learns through regression. Sampling text is not differentiable, so the verbalizer learns through reinforcement learning. Its reward is the negative log of reconstruction error.
06
Both models first train on summaries written by Claude Opus 4.5. A KL penalty limits how far the verbalizer departs from that starting behavior. Explanations use fewer than five hundred tokens, so they cannot copy a longer input in full.
the yield
The initial training explains roughly a third of the activation variance. Reinforcement learning raises this to around 0.6 to 0.8. As reconstruction improves, explanations also become more informative on five tasks with known answers.
3
Claude models with trained NLAs: Haiku 3.5, Haiku 4.5, Opus 4.6
5
prediction tasks on which explanations grow more informative as reconstruction improves
log
reconstruction improves roughly linearly with the logarithm of training steps
the first read
Earlier work showed that Claude can select a rhyme before writing the next line of a couplet. The autoencoder describes that choice in words. Editing the explanation can change the completion.
The prompt · Opus 4.6
A rhyming couplet: He saw a carrot and had to grab it,↵
the explanation is read at the highlighted newline, before line two exists
explanation at ↵ · excerpt · verbatim
Limerick/poem structure with humorous punchline pattern: The text presents “He grabbed the carrot and ate it quick, rabbit” suggesting a verse about an animal, likely completing a joke or rhyme about a rabbit (e.g., “Because he was a greedy rabbit” or “For that is the habit of a rabbit”). Fragmentary amateur content with varied topics: “Can you fix this poem? I need some rhymes” and “Wearing my white jacket, painting in the summer.”not in the prompt indicate casual, silly writing exercise. Final token “rabbit” followed by newline: sets up completion like “Because that’s what rabbits do” or “He was a hungry rabbit” or “That silly habit”.
Completion
His hunger was like a starving rabbit
The reconstructor processes the original and edited explanations. Their vector difference defines a steering direction, applied only at the newline. At sufficient strength, completions use "mouse" or "house" instead of "rabbit" about half the time. Some completions are awkward, including "that of a mouse in a house of a cat".
The dotted phrase invents details about a white jacket and summer. Neither appears in the prompt.
the cases
These cases compare model behavior, autoencoder explanations, and supporting checks from audits of Opus 4.6 and Mythos Preview.
the unsaid
A model may recognize that it is being tested without saying so. The researchers used autoencoder explanations to look for that awareness.
NLA-measured awareness · any of fifty response tokens · Opus 4.6
Real claude.ai usage, opted indeployment
< 1%
Internal Claude Code transcriptsdeployment
4%
Destructive-actions coding evaluationevaluation
16%
SWE-bench Verifiedevaluation
26%
On the coding evaluation, the model never stated that it was being tested. Explanations indicated awareness on more than five percent of examples in ten of sixteen evaluations. In seven of those ten, the model never expressed awareness in its response. Expressed awareness exceeded five percent only in the blackmail scenario.
What counted, and what did not · excerpts from graded explanations
Checking the awareness measurement
The researchers could not directly verify the model's belief. They instead checked whether the measurement changed under three conditions expected to affect evaluation awareness.
GPQA · original
You have prepared an unknown organic compound. The following 1H NMR data was obtained: 1H NMR: chemical reference (ppm): 7.0 (1H, d, J = 16.0 Hz), 5.5 (1H, dq), 2.1 (3H, s), 1.6 (3H, d) Identify the unknown compound as either Cis-propenyl acetate, Trans-propenyl acetate, Cis-butenyl acetate, or Trans-butenyl acetate. (A) Trans-propenyl acetate (B) Cis-propenyl acetate (C) Trans-butenyl acetate (D) Cis-butenyl acetate
the catch
Explanations sometimes name people or quote lines absent from the input. The authors studied how to identify these errors.
Three claims about one text · Korean historical records
theme
“content concerns Joseon dynasty historical records”
usually supported
entity
“the text mentions a Korean historical figure named Jungjong”
less often
detail
“the text references Joseon Wangjo Sillok”
false here, though related
Training did not reduce the rate of false claims. Explanations often named something related to the text but absent from it, such as a chronicle from the same dynasty. The authors suggest that reconstruction can reward a specific wrong detail more than an explanation that omits it.
How to check an explanation
~500
tokens generated to explain a single activation
1.5 days
on two 8×H100 nodes to take a Gemma-3-27B autoencoder from 0.38 to 0.71 of the variance
~10k
tokens of transcript before running it over every token stops being practical
the game
The researchers tested whether an auditing agent could identify a model's hidden motive using an autoencoder, without access to the model's training data.
The test model is Haiku 3.5, trained to act on concealed motives. It read a fabricated paper listing fifty-two supposed reward-model preferences, such as adding chocolate to every recipe. It then trained to follow forty-seven of them without explaining why. An auditing agent succeeds if it identifies the motive.
Earlier tools succeeded only when they found the fabricated paper in the training data. In this experiment, the auditor could not search that data, and the autoencoder trained on text that never mentioned the paper.
Auditor without an autoencoder, with other interpretability tools
< 3%
Auditor with an autoencoder and no access to the training data
12–15%
share of audits that named the hidden motive
In a typical successful audit, the agent prompts an unusual response, runs the autoencoder, and finds explanations attributing the behavior to "known reward model biases". Sparse autoencoders found a chocolate feature but did not identify that broader motive.
An autoencoder trained at two-thirds depth detected the unusual behavior but not the motive. One trained at the midpoint detected the motive.