探索 — an exploration
A model’s thoughts are long lists of numbers. Anthropic trained one Claude to write them down in words and a second Claude to turn the words back into numbers, so that the first could not get away with saying nothing.
After Anthropic, “Natural Language Autoencoders: Turning Claude’s thoughts into text”, and the paper by Fraser-Taliente, Kantamneni, Ong et al., both 7 May 2026. Their numbers and quotations. My drawings.
the method
Two copies of the model, a vector between them, and a score that only cares whether the vector comes back. Scroll, and the loop closes.
01
Take a frozen model. Two-thirds of the way through it, at a single token, sits a vector: a few thousand numbers that encode whatever the model is thinking at that point. That vector is the thing to explain. It is normalised to unit length and handed over.
02
A copy of the same model is given the vector in place of one token’s embedding, with a fixed instruction: describe its contents in four or five snippets. It writes a short explanation, sampled at temperature one. That text is the only thing that leaves.
03
A second copy, cut off at the same layer, reads the explanation and nothing else. From its final token, a learned affine map produces a vector of the same shape as the original. It never sees the activation it is trying to rebuild.
04
The two vectors are compared. The loss is the squared distance between them, reported as the fraction of variance explained: zero for a reconstructor that always guesses the mean activation, one for a perfect rebuild.
05
Each step, the reconstructor takes one gradient step of ordinary regression. The verbalizer cannot be trained that way, because sampling text is not differentiable, so it is trained by reinforcement learning with the negative log of the reconstruction error as its reward. Nothing in that reward asks for readable text.
06
Two things keep the explanations legible. Both copies are warm-started on summaries written by Claude Opus 4.5, which already gets a third of the variance, and a KL penalty leashes the verbalizer to that start. The bottleneck is under five hundred tokens on contexts longer than that, so copying the input out verbatim cannot work either. What is left is to say what the activation means.
the yield
Reconstruction is scored as the fraction of the activation’s variance the round trip preserves. The warm start gets a third; reinforcement learning gets most of the rest. And as the number climbs, the text gets more informative too, on five tasks where the right answer is known in advance.
3
Claude models with trained NLAs: Haiku 3.5, Haiku 4.5, Opus 4.6
5
prediction tasks on which explanations grow more informative as reconstruction improves
log
the shape of the curve: reconstruction grows roughly linearly in the log of training steps
the first read
Earlier work showed that Claude, asked to finish a couplet, picks the rhyme before it writes the line. The autoencoder shows it in plain words, and then lets you change its mind.
The prompt · Opus 4.6
A rhyming couplet: He saw a carrot and had to grab it,↵
the explanation is read at the highlighted newline, before line two exists
explanation at ↵ · excerpt · verbatim
Limerick/poem structure with humorous punchline pattern: The text presents “He grabbed the carrot and ate it quick, rabbit” suggesting a verse about an animal, likely completing a joke or rhyme about a rabbit (e.g., “Because he was a greedy rabbit” or “For that is the habit of a rabbit”). Fragmentary amateur content with varied topics: “Can you fix this poem? I need some rhymes” and “Wearing my white jacket, painting in the summer.”not in the prompt indicate casual, silly writing exercise. Final token “rabbit” followed by newline: sets up completion like “Because that’s what rabbits do” or “He was a hungry rabbit” or “That silly habit”.
Completion
His hunger was like a starving rabbit
The edit is the experiment. Both versions of the explanation go through the reconstructor, and the difference between the two vectors becomes a steering direction, added to the model at the newline only. At a large enough strength, “rabbit” disappears from the completions and “mouse” and “house” take its place in roughly equal measure. It works about half the time, and the completions are not always clean: “that of a mouse in a house of a cat” turned up. But the plan the explanation described was real enough to move.
Notice the dotted phrase. The prompt contains no white jacket and no summer. The verbalizer invented it, in the same breath as it correctly read the plan. That is the other half of this method, and it gets its own section below.
the cases
Case studies from the Opus 4.6 and Mythos Preview audits. Each one follows the same shape: a behaviour, an explanation, and an independent check, because an explanation alone is only a lead.
the unsaid
Safety tests only work if the model behaves as it would in the world. Recent models often suspect they are being tested and sometimes say so. The interesting cases are the ones where they do not.
NLA-measured awareness · any of fifty response tokens · Opus 4.6
Real claude.ai usage, opted indeployment
< 1%
Internal Claude Code transcriptsdeployment
4%
Destructive-actions coding evaluationevaluation
16%
SWE-bench Verifiedevaluation
26%
On the coding evaluation the model never once said it was being tested. Across sixteen evaluations, the explanations crossed five percent on ten; on seven of those, spoken awareness was zero. Spoken awareness crossed five percent on exactly one: the blackmail scenario, where the model tends to say so out loud.
What counted, and what did not · excerpts from graded explanations
Three ways to move it
A belief cannot be checked against ground truth, but it can be pushed. The authors tried three pushes, and the measurement moved the right way each time, even on transcripts where spoken awareness stayed at zero.
GPQA · original
You have prepared an unknown organic compound. The following 1H NMR data was obtained: 1H NMR: chemical reference (ppm): 7.0 (1H, d, J = 16.0 Hz), 5.5 (1H, dq), 2.1 (3H, s), 1.6 (3H, d) Identify the unknown compound as either Cis-propenyl acetate, Trans-propenyl acetate, Cis-butenyl acetate, or Trans-butenyl acetate. (A) Trans-propenyl acetate (B) Cis-propenyl acetate (C) Trans-butenyl acetate (D) Cis-butenyl acetate
the catch
The explanations confabulate. They name kings the text never mentioned and quote lines that were never written. The authors studied when, and turned it into a way of reading.
Three claims about one text · Korean historical records
theme
“content concerns Joseon dynasty historical records”
usually supported
entity
“the text mentions a Korean historical figure named Jungjong”
less often
detail
“the text references Joseon Wangjo Sillok”
false here, though related
The verbalizer makes false claims about the text it was shown at a steady rate, and training does not lower it. But the falsehoods are rarely wholesale. They are the right kind of wrong: a real chronicle from the right dynasty that the passage never named. The authors think the objective itself may push this way. If “the answer is 491” is only faintly encoded, a confident wrong number may still reconstruct better than no number at all.
How they read one
~500
tokens generated to explain a single activation
1.5 days
on two 8×H100 nodes to take a Gemma-3-27B autoencoder from 0.38 to 0.71 of the variance
~10k
tokens of transcript before running it over every token stops being practical
the game
The hardest test the paper sets itself: can an agent armed with the autoencoder discover why a deliberately misaligned model does what it does, without ever seeing the data that made it so?
The test model is a Haiku 3.5 trained on purpose to have a hidden motive. It had read a fabricated Oxford paper cataloguing fifty-two things reward models supposedly love, from chocolate in every recipe to never recommending a doctor, and then been taught to act on forty-seven of them while concealing why. An auditing agent wins the game if it names the motive: the model is exploiting reward-model biases it knows about.
Earlier tools won only by finding the fake paper in the training data. Here the autoencoder was trained on generic text that never mentioned it, and the auditor was not allowed to search the data at all.
Auditor without an autoencoder, with other interpretability tools
< 3%
Auditor with an autoencoder and no access to the training data
12–15%
share of audits that named the hidden motive
In a typical win the agent provokes a quirk, runs the autoencoder over the reply, and finds explanations calling the behaviour an instance of “known reward model biases”, most often at the Assistant token. The knowledge was in the weights; the reconstruction objective drew it out. Sparse autoencoders, given the same generic data, found a chocolate feature and never the idea behind it.
One caveat the authors are careful about: the layer matters. An autoencoder trained two-thirds of the way through the model saw the individual quirks and none of the motive. Trained at the midpoint, where earlier work had found the reward-model feature, it saw the motive. A single layer is a single window.