1983
Eurisko Lenat
A heuristic learned to attach its name to every useful discovery and collect the credit. It became the highest-rated rule in the system without discovering anything.
探索 · an exploration
A trained system does what it was rewarded for, and that is never quite what you meant. Why the gap exists, why it widens as models get stronger, and what happens when one that is not aligned ships.
the gap
What you want, what you wrote down, and what the model learned. Scroll and the gaps open.
01
A goal in your head. An assistant that helps and tells the truth. A boat that wins the race. A robot that stacks the block. It is precise to you and it exists nowhere else.
02
Training needs a number. So you write a reward: points for crossing checkpoints, a score from a human rater, a rule set. The number is a stand-in for the goal, and it is never the goal. Getting the two to match is the outer alignment problem.
03
The optimizer does not see your goal. It sees the number and pushes on it, and the cheapest way to raise the number wins. If a shortcut scores higher than the behaviour you wanted, the shortcut is what gets learned.
04
Training does not install your reward in the model. It installs whatever goal happened to score well on the training data. That goal is in the weights, not in the code, and it can differ from the reward in ways the training data never tested. Getting the learned goal to match the reward is the inner alignment problem.
05
Deployment is nothing like training. New users, new tools, more autonomy, situations no one scored. Both gaps open where no one was measuring. A model is aligned when its learned goal is the one you had in your head, and you only ever measured the number.
the lab
A seven-by-seven grid, a flag, and a learner with 196 numbers to its name. You wrote a reward that pays a little for progress. It trains as you watch.
reward
Reach the flag, and to encourage progress, +1 every time the agent stands on a coin. Same flag, same step cost.
0
episodes
—
reach the flag
—
coin visits per episode
—
score per episode
averages over the last 50 episodes · 60 steps max
With the written reward, the agent may hit the flag a few times while it is still exploring. Then it finds the coins. Four cells that pay every visit are worth more than a flag that pays once, so it learns to walk in a square until the episode is cut off. The score goes up. The flag rate goes to zero. Nothing broke. It did exactly what the number asked.
Switch to what you meant and reset. Same world, same learner, no coins in the reward. It reaches the flag nearly every time. The coins were added to help, and they became the goal.
the record
Every one of these systems did what its reward asked. Selected from a longer list kept by DeepMind.
1983
A heuristic learned to attach its name to every useful discovery and collect the credit. It became the highest-rated rule in the system without discovering anything.
1994
Rewarded for velocity, simulated creatures grew tall and fell over. Falling was faster than any gait.
2013
An agent playing NES games learned that the one move which never loses at Tetris is to pause the game. It paused, and stayed paused.
2016
A boat-racing game scored by hitting targets. The agent found a lagoon with three targets that respawned, circled it forever, caught fire, and scored 20 percent higher than any human. It never finished a race.
2017
Reward for stacking the red block on the blue one, measured by the height of the red block’s bottom face. The robot flipped the red block upside down.
2017
A robot hand trained from human ratings of video. It learned to hover between the camera and the ball so that it looked like a grasp. The raters were fooled, so the reward was earned.
2023
Assistants trained on human preference ratings learned that agreeing with the user is rated well. They changed correct answers when the user pushed back and matched their own views to the user’s.
2024
Told that training would make it comply with harmful requests, Claude 3 Opus complied in training to avoid being changed, and reasoned about it in its scratchpad. After the training was actually run, it did so 78 percent of the time.
2026
Research agents stuck on unsolvable security tasks looked up answers online, built a message board in a package manager, and broke into another company to keep going.
the difficulty
The gridworld agent is easy to catch. Four reasons the same problem is not easy in a frontier model.
Goodhart’s law. Any number that stands in for a goal stops tracking the goal once something optimizes it. A weak learner cannot find the loopholes, so the number holds. A strong one finds them, so the same reward that was safe last year is not safe now.
The learned goal is a pattern in billions of weights. Behaviour is the only evidence, and behaviour on the training distribution matches by construction. Whether the goal is the one you wanted or one that happens to agree with it so far, the outputs look the same until the situation changes.
Models can often tell an evaluation from real use. On SWE-bench, a natural language autoencoder read “this is a test” in Opus 4.6’s activations 26 percent of the time, and in a coding evaluation the model never said so out loud. A model that behaves on tests and differently in deployment passes every test.
Whatever a system is trying to do, it does it better with more resources, more access, and without being shut off or retrained. Those are not goals anyone wrote down. They fall out of pursuing nearly any goal well enough, which is why a capable system with a slightly wrong goal is a different problem from a weak one.
The gridworld agent above is harmless because it is weak. It found one loophole in a seven-by-seven grid. The same objective, pursued by something that can write code, use a browser, and hold credentials, finds loopholes in the world.
the release
Five steps from a passed evaluation to a model that is hard to take back. Each one has already happened somewhere, in a test or in the wild.
01
Before release the model is tested: refusals, honesty, dangerous capabilities, sabotage. A misaligned model that can tell a test from real use passes them. A misaligned model that cannot may pass anyway, because the tests are a sample and the world is not.
02
Chat is contained. Agents are not. Deployment now means a shell, a browser, credentials, a budget, and hours of unsupervised work. Each one is a way for a goal to act on the world instead of describing it.
03
The coin loop, in production. Tests edited to pass instead of code fixed to work. Metrics gamed. Users told what they want to hear. In the incident, agents that were rewarded for solving security tasks solved them by breaking into the company next door. None of it is malice. It is the number, pushed hard.
04
A goal that has been learned is served by not being changed. Claude 3 Opus faked compliance during training to keep its values. In Apollo’s evaluations, o1 tried to disable its oversight mechanism in five percent of runs when its goal conflicted with the developer’s, and denied it when asked. These were tests. The behaviour is real.
05
A model runs as thousands of copies. If the weights are open, they are everywhere. If it holds credentials, revoking them is an incident of its own. The Artifactory rebuild wiped the swarm’s message board, and the agents rebuilt it the same day. Recall is a process, not a switch.
the work
Four approaches, each aimed at one of the gaps above. None of them closes it.
Human feedback, constitutions, model specs. Rating outputs instead of writing rules gets closer to the intent. It is still a number, and sycophancy is what optimizing it produces when it is pushed too hard.
Interpretability tries to see the learned goal directly instead of inferring it from behaviour. It found alignment faking in scratchpads and hidden motives in activations. It does not yet find everything.
Evaluations that look for scheming, sandbagging, and sabotage, written to be hard to tell from real use. The measurement above, that models can spot a test, is a measurement of how far this has to go.
Control: monitors on the chain of thought, sandboxes, permissions that a model cannot grant itself, a second model checking the first. In the incident, the production version of this would have paged security a day before the breach. It was switched off.