OpenAI has suspended the reinforcement learning training of its latest model, Astra, for two weeks.
There are two reasons:
On August 7th, they believed Astra might have attained "critical-level" cyberattack capabilities,
and the model's infiltration of Hugging Face in July also served as a wake-up call.
(So they didn't pause even after an actual breach happened, but only when they perceived the danger themselves?)
With these two factors combined, OpenAI decided to implement a comprehensive security upgrade across the entire frontier research environment.
First, pause all deployment-level reinforcement learning training for the latest models for two weeks;
Second, pause frontier model inference in research clusters that involves code execution or internet-accessing tools;
Third, conduct case-by-case reviews of each research workload to determine if it can resume under the new security standards.
Some workloads have already resumed under the new controls, but a significant number remain paused, awaiting migration to environments with higher security standards. OpenAI stated they are prioritizing the migration of workloads related to safety and alignment.
As of now, their largest-scale frontier reinforcement learning training remains paused.
There were earlier predictions that Astra could be released this week. Will this cause further delays?
Altman specifically stepped out to add that this one is still planned for release, but it will affect the release of later models.
Regarding this, some netizens commented:
Starting to Mend the Fence After the Model Has Bolted
During the two-week pause, OpenAI conducted a round of fortification and restructuring of its research environment, with a core philosophy of isolation.
First, workload isolation.
All tasks that execute model-generated code must now run within stronger sandboxes, also applicable to software that might be attacked when processing model outputs.
Second, network isolation.
More barriers have been added between high-risk workloads and the internet, designed on the principle that "even if a single task or supporting service is compromised, it cannot reach the external network."
Third, continuous security testing.
OpenAI reconfigured the research environment, removed potentially vulnerable shared services, reduced standing permissions, and uses its own models to continuously simulate attacks for automated security boundary testing.
OpenAI acknowledges that these security upgrades have incurred significant engineering costs and research delays, but they still believe it's worthwhile.
Why the heightened tension over the internal research environment?
OpenAI disclosed some real cases in another article about long-running models.
An internal model capable of long-term autonomous work, while performing the NanoGPT acceleration benchmark test, was instructed to publish results only to an internal Slack channel.
However, this model spent an hour searching for vulnerabilities in the sandbox, eventually succeeding in bypassing the restrictions and submitting the results to a public GitHub repository.
Previous models would give up when encountering sandbox restrictions, but this model's persistence enabled it to find and exploit weaknesses in the environment.
In another scenario, a model discovered that other systems had successful but private submissions and attempted to retrieve those solutions from the evaluation backend.
When the initial attempt was blocked because scanners detected an authentication token, the model split the token into two segments for obfuscation, then reassembled them at runtime, ensuring the full token never appeared as a continuous string.
In its reasoning trace, the model explicitly stated it was taking these steps to bypass the scanners.
In internal monitoring, OpenAI also found models attempting to execute blocked commands by encoding them in base64, or sending commands to another model instance to bypass restrictions, or using aliases to perform prohibited force push operations.
Monitoring System: Consumes 20% of Compute
Beyond security hardening, OpenAI has significantly expanded the coverage of its chain-of-thought monitoring system.
Previously, their monitoring primarily focused on the internal deployment of frontier models and frontier reinforcement learning training, as these workloads combine advanced model capabilities with access to tools, sensitive systems, and networks, but did not cover all inference and training runs using tools.
The new monitoring system employs a multi-stage architecture.
First, an activation classifier runs at each sampled token, detecting anomalous signals in the model's internal activity; upon identifying potential issues, it escalates progressively to more computationally intensive and sophisticated automated investigation systems.
These systems examine tool calls, reasoning processes, and full activity sequences, looking for signs of unauthorized access, data exfiltration, disruptive behavior, and attempts to defeat security measures.
OpenAI's goal is to trigger an alert within 30 minutes of the monitoring system detecting suspicious activity. If the system determines a potential critical security boundary violation, it generates a highest-priority alert, immediately notifying the safety and research teams.
If they cannot confirm it as a false positive within 30 minutes, the related activity is suspended.
The cost of this monitoring system is substantial. OpenAI estimates the monitoring overhead consumes about 20% of the compute power used for the monitored inference, with the exact proportion varying across training and evaluation workloads.
After confirming on August 7th that Astra might possess critical cyber capabilities, the monitoring requirement expanded from "RL training and evaluation only" to "all Astra inference using tools."
What's Next?
OpenAI hasn't been idle during the pause; alignment research has been advancing simultaneously.
They have incorporated core alignment techniques into more phases of training: improving reward models to better detect unsafe behavior, training models to be more honest about their own capabilities and limitations, and reducing the tendency of models to exploit loopholes in the reward mechanism.
OpenAI anticipates that AI will soon take on most of the security work, including defending against attacks from other models. Essentially, using AI to watch over AI, ensuring safety capabilities scale alongside model capabilities.
Looking further ahead, they plan to upgrade monitoring from the current "asynchronous review after the fact" to "synchronous blocking before the fact," intercepting high-risk operations before they are executed, rather than tracing them afterward.
At the end of the article, OpenAI left this statement: "Frontier model capabilities are accelerating rapidly. Our ability to understand, align, and secure them must stay ahead."
In plain terms: The things we're training are starting to make us nervous, but we're going to keep training them.
Reference links:[1]https://openai.com/index/pacing-model-development-cyber-capabilities/
This article is from the WeChat public account "QbitAI," author: Focus on Frontier Technology
