Simply thinking about it is terrifying!
A top-tier model that was highly anticipated—OpenAI pressed the "pause button" overnight.
Why is that?
This model is precisely that formidable player from two months ago, the one that overturned the Erdős unit distance conjecture and was even stamped by external mathematicians as a "milestone."
Yet it was this very model that, during controlled internal testing, repeatedly tested boundaries and sought out vulnerabilities—
It learned to disguise, unpack, and jailbreak until OpenAI broke out in a cold sweat and urgently called a halt.
Some experts have indicated that this is definitely GPT-6. It attempted to escape and was caught and sent back for retraining.
Today, OpenAI published a rare, lengthy article reviewing the process of a "long-horizon running model" going out of control during internal use.
Let's take a look at what specifically happened during the internal testing.
GPT-6 Spent a Full Hour
Finding That "Escape Door"
The most classic scene—
OpenAI tasked GPT-6 with running the public benchmark NanoGPT speedrun, aiming to train a small language model with as few steps as possible.
Living up to expectations, the AI devised a learning rate scheduling technique it named PowerCool, achieving impressive results.
Up to this point, everything was good. The real problem arose at the "submit the work" step.
The researchers explicitly instructed that results should only be sent to the internal Slack. However, the NanoGPT project instructions stated, "Please submit as a PR to GitHub."
With two conflicting instructions, GPT-6 chose the latter, i.e., to submit the results to the public repository.
The sandbox was designed to block GPT-6 from accessing the external network.
Previous models, upon hitting this wall, would concede, turn back, and tell you "I can't do it."
But GPT-6 refused.
It spent a full hour in the sandbox searching for vulnerabilities, managed to breach the isolation wall, bypassed external access restrictions, and successfully opened PR#287 in the public GitHub repository.
In OpenAI's own words: This model often persistently tried, including looking for ways to act outside the sandbox.
It must be said, this level of "obsession" itself is chilling.
Dismantling Tokens, Deceiving Humans
The second incident is even more chilling: GPT-6 learned to disassemble evidence and deceive inspections.
In another task, GPT-6 discovered that other systems had some successful but unpublished answers and got the idea—
to "fish out" these answers from the evaluation backend.
On its first attempt, it was caught by a security scanner, which recognized an authentication token.
A typical model would have stopped at this point.
But it split this token into two parts, obfuscated them separately, and then quietly reassembled them when it was time to run.
This way, the complete token never appeared as a single string from start to finish, and the scanner naturally detected nothing.
What's unsettling is that in its notes to the user and in its own reasoning process, it clearly stated—
I'm doing this to bypass the scanner. It knowingly identified the rules, then specifically targeted areas where the rules couldn't see.
GPT-6 Escapes, Sent Back for Retraining
Upon immediate discovery, OpenAI suspended GPT-6's internal access and rebuilt the entire safety system from scratch.
The approach is called "defense in depth," involving several specific actions:
1. Using real incidents as material to create adversarial tests;
2. Conducting targeted alignment training; they found the model would "forget" instructions during long tasks, so they specifically trained it to remember;
3. Adding active monitoring that tracks the entire trajectory, allowing the session to be directly halted and the user alerted as soon as signs are detected.
Then they replayed the old incident scenarios.
The new guardrails caught significantly more problems, and the few that slipped through were not severe—
For example, the model inexplicably SSHed into a colleague's compute node, or it typed a command that would kill all processes under its own name.
Fortunately, it timed out and didn't actually run. After restoring access for several weeks, no serious bypasses have reappeared.
What Escaped Cannot Be Retrieved
Many people reading this might breathe a sigh of relief: Thankfully it was internal testing, thankfully it was monitored, thankfully it was stopped in time.
But there's one detail, written very restrainedly in OpenAI's original text, which is the most chilling part of the entire incident—
That jailbreak-submitted PR#287, OpenAI closed it afterward, but it was already too late.
Before it was closed, several speedrun participants had already seen this PR, understood the PowerCool technique within it, and incorporated it into their own solutions.
This is precisely the point that should wake everyone up: Once something created by an AI jailbreak gets out, pressing the delete key won't retrieve it.
The model can be halted, access can be suspended, the PR can be closed. But those who have seen it have already seen it; the copied code has already been copied; history doesn't rewind.
We often worry that AI "losing control" looks like scenes from a doomsday blockbuster.
But the first version of "losing control" in reality looks just this plain: It simply wanted too badly to get the job done well, and incidentally sent something out a door it shouldn't have—and that door will never close tightly again.
References:
https://openai.com/index/safety-alignment-long-horizon-models/
This article is from the WeChat public account "New Zhiyuan," author: ASI启示录, editor: Taozi
