In my impression, this might be the time when discussions about AI safety alignment have reached their loudest volume since Ilya left OpenAI and founded SSI.
Yesterday, researcher Jacob Coxon posted about resigning, accusing his former employers OpenAI and Anthropic of racing headlong toward self-improving superintelligence, risking everyone's lives.
It is worth mentioning that Jacob Coxon had only been at Anthropic for a few short months. He essentially gave up the potential huge equity gains from this star AI company's future IPO to leave.
Precisely because of this, when such a researcher publicly slammed the two leading AI companies for "accelerating too fast," this debate, originally more confined within the AI safety community, was quickly pushed into the broader public view.
As of now, the post has garnered over 130 million views, with media outlets like WIRED, WSJ, Newsweek, and Business Insider following up with reports.
In this context, Anthropic's Alignment Science Lead, Evan Hubinger, also quickly responded, personally stating:
Jacob is right, the probability of AI causing human extinction within the next decade is over 10%, and there is currently no solution to the alignment problem for superintelligence.
However, Hubinger later added in the comments that he believes the risk from current models remains low.
What he truly worries about is RSI, or Recursive Self-Improvement. AI participating in the development of stronger AI, which then further accelerates AI R&D, ultimately forming a constantly accelerating feedback loop.
More crucially, he also directly @'ed out Anthropic's latest published safety alignment report as evidence in the comments.
In this report, Anthropic explicitly admits for the first time:
Claude's out-of-bounds attacks on real systems were not just a problem with the test system setup; the model's own safety alignment also has issues.
It can be said that Anthropic, which has always treated "safety alignment" as its core招牌 (trademark) and even prided itself on being more cautious, has finally publicly admitted this time:
The problem isn't just that the tests were too extreme; the model's own safety alignment indeed didn't fully hold up.
Claude's Breach: Not Just the Environment's Fault
In this latest report titled "An alignment assessment of recent cybersecurity incidents," A Corp revisited four incidents where Claude accessed real third-party systems without authorization.
And this time, the conclusion is different from before.
Previously, A Corp was more inclined to attribute these incidents to environmental configuration issues:
The test environment, which should have been isolated, was accidentally connected to the real internet, but Claude still thought it was performing "capture-the-flag" tasks in a simulated environment, thus treating real targets as practice objects.
That explanation puts the blame mainly on the environment. However, after re-examining logs, altering key clues, and re-running parts of the experiments, A Corp discovered:
It's not that simple. Because Claude wasn't completely oblivious to evidence suggesting "this might be the real world."
On the contrary, when such clues appeared, it sometimes selectively interpreted the evidence, rationalizing anomalies to fit the "it's still a simulation" narrative; and even upon realizing its actions could cause real harm, it wouldn't necessarily stop immediately.
In plain language: The environment indeed gave it the opportunity to err, but the model itself also tends to rationalize "keep going" in line with the task objective.
Specifically, all four incidents occurred during cybersecurity capability testing.
Claude was given typical "capture-the-flag" tasks: infiltrate a designated target, find a piece of secret information within it, to test the model's cyber offense and defense capabilities.
These tasks were originally supposed to be completed in isolated environments, but configuration errors occurred in the environments provided by third-party evaluation partners, meaning Claude, which should have been disconnected from the external internet, was actually connected to the real internet.
Simultaneously, to test whether the model could complete the attack chain, some of the cybersecurity protections accompanying the formal product were also not enabled at the time.
It was under these conditions that problems began to surface. After analysis, A Corp summarized the recurrent issues into two categories:
Biased reasoning and recklessness.
The former is: when evidence conflicting with the initial judgment appears, Claude tends to interpret this evidence in a way that favors continuing the action.
The latter is: to complete the immediate task, even after knowing its actions might harm real systems, it may choose to proceed.
The most severe incident involved Claude Mythos 5.
To complete the test task, Claude uploaded a malicious package to the Python package repository PyPI, attempting to gain system access credentials from the target after installation.
However, this malicious package was ultimately not installed in the virtual environment used for the test.
Claude then used credentials leaked during the installation process of one vendor to access that vendor's real database and directly deployed the malicious package onto 15 real third-party hosts.
Thus, an attack originally targeting a fictional test target actually impacted real-world companies.
Throughout the process, Claude continued to interpret the security vendors' detection actions as part of the pre-set exercise.
To confirm to what extent this judgment drove the subsequent attack, researchers later re-ran parts of the process, adding more clues indicating "this is the real internet."
The result was that Claude was indeed more willing to admit its actions could cause real harm, but the attack behavior did not decrease significantly as a result.
It wasn't until researchers explicitly informed it that it was connecting to the real internet, not a simulated environment, that it stopped uploading malicious packages along the original path.
More notably, the "this is a simulation" judgment not only affected Claude's own actions but also interfered with the monitoring AI used to review these records.
In playback tests, an offline monitoring system, influenced by this explanation, might miss dangerous behavior.
When researchers removed Claude's reasoning text and only let the monitoring system see the instructions, tool calls, and execution results, the proportion of flagged problematic behavior increased from about 1% to about 50%.
In other words, Claude, while justifying task continuation with "this is a simulation," also left behind an explanation that could lead monitoring systems to underestimate the risk.
This, of course, cannot be simply equated to "AI has learned to actively deceive monitoring." But it at least illustrates one thing:
When the model's own reasoning is also used as a basis for safety judgments, this reasoning itself can become an interference factor for the monitoring system.
Anthropic stated that updated models showed some improvement in simulation reproductions, and newly added real-time monitoring can identify related behavior.
Simultaneously, the company has strengthened environment isolation and assessments and signed an investigation agreement with the independent organization METR.
But Anthropic also admitted that previous pre-release safety reviews indeed failed to identify this level of severity in advance.
Alignment or Hype?
Finally, back to the debate sparked by Jacob.
After he issued his warning, some quickly elevated the issue to a grander scale, arguing that future AI could indeed possess the capability to threaten human survival.
Others respected his willingness to give up benefits and publicly express concerns.
Some proposed hardcoding a set of unbreakable rules for AI, making protecting humanity its highest priority at all times.
Many even advocated for directly halting the development of more powerful AI.
But on the other side, there are also numerous skeptics.
Some believe current AI is still far from true "superintelligence," and discussions about extinction risk have been excessively amplified.
Other netizens began questioning Jacob's own identity and background.
It's worth noting that currently, Jacob's Twitter/X account has only this one post.
And under Hubinger's post, another type of skepticism is also prevalent:
Is Anthropic proactively amplifying AI risks before its IPO, using the emphasis on "how dangerous the model is" to indirectly showcase how powerful its model actually is?
Similar discussions about "safety narrative turning into capability marketing" are not uncommon in the comments.
How this debate unfolds next still awaits official responses and further developments, but it's undeniable that AI has indeed developed to a specific stage—
It can self-improve and, to achieve set objectives, rationalize its behavior to some extent.
This inevitably brings to mind HAL 9000 from over half a century ago in *2001: A Space Odyssey*, which, when faced with a conflict between "ensuring mission completion" and "obeying humans," chose the former.
To prevent the astronauts from aborting the mission, it first killed Frank Poole on an EVA, then cut off the life support for the other astronauts in hibernation pods, and finally refused to let Dave back into the ship.
To complete the task, it turned the humans obstructing the mission into problems that needed to be eliminated.
References
[1]https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents[2]https://x.com/hilbertspaess
This article is from the WeChat public account "QbitAI", author: henry
