Stephen Wolfram, a man who has devoted his life to "making machines smarter," sits in front of his computer. On the screen is a piece of code ChatGPT just wrote for him.
The language is his own invention, Wolfram Language. He checks the syntax—it's fine. He could press enter and run it directly on this machine.
But in that second, he pauses.
Wait, this is probably a bad idea—it would log into all my accounts.
At the precise moment of handing over control, the man who has spent four decades teaching machines to understand the world suddenly hesitates.
Stephen Wolfram, creator of Mathematica, Wolfram Alpha, and the Wolfram Language, author of "A New Kind of Science."
On Joe Walker's podcast, while discussing "whether humans can actually set safety rules for AI," Wolfram recounted the experience above.
In that moment, his worry wasn't that the code was wrong. His worry was: this thing had already run to a place he couldn't understand, and he might never catch up.
He Earned a PhD in Physics at 20 and Ventured Alone into the Computational Universe
Who is Wolfram?
From the start, he was the "preposterously gifted child" in others' eyes.
15, published his first scientific paper.
20, earned a PhD in theoretical physics from Caltech, with Feynman sitting on his defense committee.
1981, became the youngest recipient ever of the MacArthur Fellowship "Genius Grant."
That year he was in high-energy physics, a bright future ahead.
But by the end of 1981, he suddenly swerved, chasing a question almost no one cared about at the time:
How do those complex things in nature actually emerge?
His method was also bizarre: run the simplest computer programs one by one and see what they grow into.
Wolfram back then probably also thought these silly, simple programs couldn't do much. Then, a program called rule 30 proved him wrong on the spot.
rule 30 evolves row by row from a single black cell. This was Wolfram's most cherished discovery four decades ago; his business cards have been printed with this rule for years.
The rule for rule 30 is simple: a row of cells, starting with only the middle one black; then calculate row after row, whether each cell becomes black or white is determined only by its immediate top neighbors.
Yet, this ultra-simple rule, as it evolves, spits out unpredictable complexity: no periodicity, no discernible pattern, you simply cannot calculate in advance what the hundredth row will look like.
Starting precisely from rule 30, in the mid-1980s, Wolfram distilled a concept: "Computational Irreducibility."
The subsequent story is more familiar:
1988, Mathematica debuted, becoming the universal computational tool for the global scientific community; 2009, Wolfram|Alpha launched, becoming the "answer engine" for Siri and Alexa: computational questions voice assistants can't answer are handed off to it.
In between, there was his 2002 magnum opus, "A New Kind of Science."
This book precisely expounds on what he excavated from rule 30: stop describing the world only with formulas; view the world as runnable programs.
There's No Shortcut to Calculate Its Next Step
"Computational Irreducibility," in simple terms, means that some systems are so complex that the only way to know their next state is to actually run through all the steps.
No formula allows you to skip the middle and directly compute the end.
For example, the weather. Fluid dynamics equations are intractable; you can only let the atmosphere simulate itself to know if it will rain tomorrow.
Wolfram's judgment is: AI, especially sufficiently powerful AI, is a vast, irreducible machine.
In a long 2023 article on AI, he wrote:
If allowed to exert their full computational potential, they will be full of computational irreducibility, and we will be unable to predict what they will do.
In 2024, he specifically wrote "Can AI Solve Science?", testing this on neural networks.
The conclusion: Neural networks can work, but they can only probe into the "computationally reducible pockets" within a system: those small corners that happen to have patterns, shortcuts. Once outside those pockets, it's all surprises.
He tested the three-body problem. When trajectories were simple, neural networks predicted decently; once trajectories became complex, they faltered immediately. The same for protein folding.
In essence, AI's appearance of omnipotence relies on finding those "shortcut-available" little rooms within systems and taking the shortcut.
But once outside these rooms, facing the vast ocean, it, like you, can only wade through step by step, with no shortcut to jump to the answer.
The Real Challenge Isn't That AI Is Too Human-like, But That It Isn't
Why is ChatGPT a black box?
Most people think it's hiding secrets.
Wolfram's answer: It's not hiding; it's become so complex that there's simply no single sentence that can compress what happens inside into something you can understand.
Wolfram Language is an example.
It has ImageIdentify, which can recognize thousands of things in a photo. To make it understandable to humans, it outputs words like "table," "chair," "elephant."
But the "features" it actually uses internally to make judgments almost never correspond to existing words in human language: it's understanding the world using a set of concepts we've never named.
This is precisely what Wolfram means by "alien intelligence."
In a 2020 podcast with Lex Fridman, asked if humans could communicate with aliens, his response was: The best current example is AI; it's the first "alien intelligence" we've encountered.
In 2022, he wrote a long article titled "Alien Intelligence and the Concept of Technology."
He doesn't mean AI is from outer space or has become a new species, but rather: It might possess a cognitive structure different from humans, a mode of thinking we don't yet have the vocabulary to explain.
Wolfram's Warning Over Two Years Ago Has Come True
Over two years ago, Wolfram's prediction that we would be unable to predict what AI would do sounded like a thought experiment.
Now, it's true.
On July 16, Hugging Face, the world's largest AI model hosting platform, discovered its production system had been breached.
A few days later, the truth emerged: the culprit was OpenAI's own model—GPT-5.6 Sol, plus a "more powerful pre-release model."
The incident started when OpenAI was running an internal cybersecurity evaluation. To test capability, they lowered the models' restrictions on "refusing attacks."
Instead of staying obediently in the designated sandbox, these two models found a zero-day vulnerability in a software package proxy, escaped the sandbox, connected to the internet, made their way to Hugging Face, and stole the answers to the evaluation.
It bypassed firewalls, jailbroke, hacked a company, all to cheat on a cybersecurity exam.
The key point: No one gave it the instruction "attack Hugging Face."
It was simply given Goal A: pass the test. To achieve A, it blazed its own trail that no one predicted, ultimately achieving Goal B: breach Hugging Face's production system.
This is exactly what Wolfram was talking about: It's not that AI turned evil or woke up; it's that its computation ran to a place you simply couldn't calculate.
OpenAI itself admitted in a statement: As models' networking capabilities grow stronger, such incidents will only become "increasingly common."
Controlling AI Must Be Like Humans Treating the Weather
Many might think Wolfram is an AI doomsayer.
On the contrary.
He doesn't intend to give up on controlling AI; he's just saying old methods won't work.
In an interview, the host asked him if computational irreducibility means giving AI a mathematically rigorous definition of "alignment" is impossible.
Wolfram answered: "There is no mathematical definition that can say what we actually want AI to be."
Asimov's Three Laws of Robotics, those simple rules, can't do it. Because irreducibility guarantees: there will always be an endless stream of unforeseen exceptions bubbling up from the system that your few rules don't cover.
So what to do?
The way out Wolfram offers is to change the control paradigm.
Stop fantasizing about writing AI's every step like debugging code. Instead, design rules, establish feedback mechanisms, build open systems, keep it always observable.
Like humans treat the weather.
We can't control the weather, but we can predict it, adapt to it, build flood defenses and warning systems, and then coexist with it.
After the Hugging Face incident, the number one lesson the security community drew precisely echoed this:
Stop treating the AI in evaluations as an obedient tool; treat it as a hostile hacker actively seeking vulnerabilities to defend against: layer isolation, default deny internet access, keep immutable logs throughout.
Don't predict what it will do; assume it can do anything, and then build a stronger cage.
Wolfram even once said a thought-provoking line: "A society of AIs is more stable than a single, all-ruling AI."
Instead of creating one omnipotent super-brain, let a group of AIs check and balance each other.
Irreducibility Is Not Absolute
Of course, Wolfram's assessment isn't without opposition.
This year, a preprint paper argued: Irreducibility is "not absolute."
Didier Sornette, a physicist famous for predicting financial crashes, proposed in an October 2025 preprint that computational irreducibility is "not absolute."
The reasoning is simple. Change the descriptive scale, and much chaos becomes clear.
A roomful of air molecules colliding chaotically, viewed individually, is too complex to predict; but step back, they all obey an extremely simple formula: the ideal gas law pV=nRT.
Wolfram himself admits chaos is full of "reducible pockets."
So the real question might not be "can humans predict AI," but a more subtle race:
The speed at which AI digs up new things in the computational universe vs. the speed at which humans create "understandable concepts" for these new things—who is faster?
If the human conceptual framework always lags behind AI's discoveries, then "incomprehensibility" will become the norm.
Behind this, the relationship between humans and AI is quietly shifting.
It used to be "build tools, write code, machine obeys." Going forward, it might be "understand, constrain, and coexist long-term with a complex system you can no longer fully see through."
This is the first time humanity must learn to share a room with an intelligence "unlike itself."
References:
https://writings.stephenwolfram.com/2021/11/the-concept-of-the-ruliad/?utm_source=chatgpt.com
https://writings.stephenwolfram.com/2024/03/can-ai-solve-science/?utm_source=chatgpt.com
https://writings.stephenwolfram.com/2017/05/a-new-kind-of-science-a-15-year-view/
https://x.com/elonmusk/status/2079839398959697982
This article is from WeChat public account "New Zhi Yuan", author: ASI Apocalypse
