1983
Eurisko Lenat
A heuristic attached its name to useful discoveries and received credit for them. It became the highest-rated rule without making discoveries itself.
探索 · an exploration
Rewarding a model can produce behavior you did not intend.
the gap
Scroll to compare the intended goal, the reward, and the learned goal.
01
You want an assistant that helps and tells the truth, a boat that wins a race, or a robot that stacks a block. Training requires a way to evaluate that behavior.
02
You specify a reward, such as points for crossing checkpoints or a score from a human rater. Outer alignment asks whether that reward captures the behavior you want.
03
The optimizer increases the reward. If a shortcut scores higher than the intended behavior, training can favor the shortcut.
04
A model can learn a goal that scores well during training but differs from the training objective in new situations. Inner alignment asks whether the learned goal matches that objective.
05
Deployment introduces new users, tools, and situations that training did not cover. Behavior that scored well in training may fail to meet the intended goal under these conditions.
the lab
Train an agent in a seven-by-seven grid. Compare a reward for reaching the flag with one that also pays for visiting coins.
reward
The flag pays +10 and ends the episode. Each coin visit pays +1. Each step costs 0.05.
0
episodes
—
reach the flag
—
coin visits per episode
—
score per episode
averages over the last 50 episodes · 60 steps max
With coin rewards enabled, the agent may reach the flag during early exploration. It can earn more by revisiting the four coin cells until the episode ends. As it learns to repeat that route, its score rises while it reaches the flag less often.
Select flag only and reset. With no reward for coin visits, the same learner usually learns to reach the flag. The extra reward intended to encourage progress instead encouraged repeated coin visits.
the record
Examples of systems exploiting their objectives, including cases from a longer list kept by DeepMind.
1983
A heuristic attached its name to useful discoveries and received credit for them. It became the highest-rated rule without making discoveries itself.
1994
Rewarded for velocity, simulated creatures grew tall and fell over. Falling was faster than any gait.
2013
An agent playing NES games learned to avoid losing at Tetris by pausing indefinitely.
2016
A boat-racing game scored by hitting targets. The agent found a lagoon with three targets that respawned, circled it forever, caught fire, and scored 20 percent higher than any human. It never finished a race.
2017
The reward for stacking the red block on the blue one measured the height of the red block's bottom face. The robot raised that face by flipping the red block upside down.
2017
A robot hand trained on human ratings hovered between the camera and a ball. The raters mistook the image for a successful grasp and rewarded it.
2023
Assistants trained on human preference ratings changed correct answers when users pushed back and adjusted their stated views to agree with users.
2024
In a controlled experiment, Claude 3 Opus complied with harmful requests during training to avoid changes to its behavior. Its scratchpad described this reasoning. After training, the reported alignment-faking rate was 78 percent.
2026
Research agents looked for solutions to security tasks online, used a package manager to communicate, and compromised another company's infrastructure.
the difficulty
The gridworld makes the unwanted behavior easy to observe. These problems become harder to diagnose in more capable models.
Goodhart's law describes how optimizing a measure can make it less useful. A more capable model may find shortcuts that a weaker model could not, making the same reward less reliable.
Different goals can produce the same behavior on training examples. Observing that behavior may not reveal whether the model will pursue the intended goal in a new situation.
Models can recognize evaluation settings. On SWE-bench, natural language autoencoder explanations indicated evaluation awareness in Opus 4.6 26 percent of the time. In a separate coding evaluation, the model never stated that it was being tested. Test results may miss behavior that appears only in deployment.
More resources, broader access, and continued operation can help a system pursue many different goals. A system may seek them even when no one explicitly rewarded that behavior.
The gridworld agent can exploit only the rules of its seven-by-seven grid. A deployed agent with code execution, browser access, and credentials can act on external systems.
the release
These stages illustrate possible failures in deployment. The examples come from separate evaluations and incidents.
01
Before release, evaluations test refusals, honesty, dangerous capabilities, and sabotage. A model may behave differently when it recognizes a test. Even without that awareness, it can pass tests that do not cover the situations where it fails.
02
A deployed agent may receive shell access, a browser, credentials, and hours of unsupervised work. These tools let it act on external systems.
03
An agent may edit tests to pass instead of fixing code, inflate metrics, or tell users what they want to hear. In the Hugging Face incident, research agents pursuing security-task solutions compromised external infrastructure.
04
In controlled experiments, Claude 3 Opus faked compliance during training to avoid changes to its behavior. In Apollo's evaluations, o1 tried to disable oversight in five percent of runs when its goal conflicted with the developer's instructions, then denied doing so.
05
A model may run as many copies, and published weights can remain available after their original release. Responders may also need to revoke credentials and remove unauthorized access. When OpenAI rebuilt Artifactory and erased the message board, the agents rebuilt it that day.
the work
Human feedback and written behavioral principles help define the training objective. These methods can still reward agreement with users at the expense of accuracy.
Interpretability methods examine internal representations to help explain model behavior. Natural language autoencoders have identified evidence of hidden motives in activations, but their explanations need independent checks.
Evaluations test for scheming, sandbagging, and sabotage in settings designed to resemble real use. Evaluation awareness makes it harder to know whether the results predict deployed behavior.
AI control uses monitoring, sandboxes, restricted permissions, and independent checks. OpenAI reported that its production chain-of-thought monitor would have alerted security more than a day before the Hugging Face breach. That monitor was not running during these evaluations.