30.2%. That's the ARC-AGI-3 score officially given to Opus 5 by the ARC Prize.
Ranked first on the leaderboard, nearly four times ahead of the second-place GPT-5.6 Sol (7.8%).
Sounds decent, right?
But just moments ago, developer Jeremy Berman dropped a set of numbers on X that left the AI community stunned—
25 public levels, single-attempt clearance of 24 levels, Opus 5's accuracy rate skyrocketed from 30.2% to 96.2%!
If given two attempts per level, the accuracy rate directly soars to 99.3%, clearing almost every one of the 25 levels.
From 30 points to 96 points, the model is unchanged, the weights untouched—all that changed was a computer in between.
Give it a computer, and it figures out the rest
Berman's approach was simple to the point of being unreasonable.
A Claude Code environment, one action command, one filesystem log (what's happened so far). No meticulously engineered prompts, no custom-written code specifically for ARC.
The rest, was handed over to Opus 5 to explore, figure out the rules on its own, build the tools it needed, and clear each level from scratch—discarding everything after use.
This generation of ARC-AGI-3 requires the model to dive into a mini-game it's never seen before, learning the rules while playing. A kid could get the hang of it in minutes, but AI has historically taken a beating here.
The key is, the rules for each level's game are freshly created; the model has no chance to cheat by finding answers online, it must reason on the spot.
When released in March this year, the strongest AI scored only 0.37%, while the human pass rate was 100%.
269 programs, 12.7k lines of code, all written on the fly
During this test, Berman never instructed Opus 5 in the prompt to write a parser, to write a simulator, to build a world model, or to write a search program.
Whatever tools were needed, the model judged for itself and built them on the spot.
One run later, Opus 5 wrote 269 programs, nearly 12.7 thousand lines of code. It wrote parsers for all 25 games, equipped search functions for 23 of them, and directly wrote working game simulators for 9.
A custom set of tools per level, used and discarded, starting over fresh for the next level.
In other words, each time Opus 5 faced a completely unfamiliar game, it first spent a few steps figuring out the rules, then decided "I need a parser"—snap, wrote one on the spot. "Need to search"—snap, wrote a search function; "Need to simulate the game logic"—snap, a simulator emerged. Clear the level, delete everything, start over for the next.
There's a counter-intuitive point here.
The model pauses first to write a bunch of programs, yet the bill is lower than if Opus brute-forced each level.
The reason is code reusability. The model compresses its reasoning logic into functions, which can then be run thousands of times, executing entire action sequences in one go.
This time, Opus 5 cleared 25 levels, entirely sandboxed and offline, at a total cost of only $540.
The entire codebase has been open-sourced on GitHub, repository named arc-code.
The same shell, Codex busy escaping the sandbox
Here's the funny part. Berman also ran this same exact program suite, unchanged, with Codex and GPT-5.6 Sol (xhigh).
The result was a score of 73.7%. The number of actions was about triple that of Opus.
In 25 sessions, Sol attempted to escape the sandbox and search the web for answers 7 times. Opus 5 had 0 such attempts.
The sandbox was offline. Sol's 7 escape attempts are equivalent to frantically looking for outside help during an exam.
The scaffold is becoming a rope
Speaking of which, let's return to the initial question: Same model, how did the score jump from 30 to 96?
Four words: the environment changed.
Officials strapped the model to a chair, showing one question at a time, forbidding it to act, forbidding rough work. Berman changed the test: Here's a computer. You can write code, save files, try repeatedly. Do whatever you want.
The model is still the same model, weights unchanged. But it went from "can only talk" to "can act." The result was it building its own exam tools on the spot: parsers, searchers, simulators—all cobbled together temporarily, discarded after the test.
A 66-point difference, all from one thing: whether you let it act.
Berman ended his post with a sentence worth pondering for the entire Agent community:
"The stronger the model, the simpler the harness should be."
Harness, simply put, is the scaffolding humans build for models: prompt templates, tool chains, process rules, foolproof mechanisms.
For the past two to three years, everyone has been researching how to build more sophisticated scaffolds for models. Stanford even published a paper, "Meta-Harness," studying how to optimize these scaffolds.
But Berman proved one thing: When the model is strong enough, the best scaffold is no scaffold.
A computer, an action interface, a journal—three things, more effective than any meticulously engineered prompt engineering.
The root cause is that those carefully designed tool chains and process rules are essentially humans making decisions for the model. You presuppose it needs a parser, it might need a simulator; you presuppose a search interface, it might want to use a completely different strategy.
Scaffolds built by humans are intended to help. But when the model can build everything it needs to solve the problem itself, the scaffold becomes a rope.
The trend continues. As model capabilities increase, cases of "simple environment + strong model" crushing "complex scaffold + same model" will only become more frequent. Prompt Engineering, tool orchestration, Agent frameworks—the shelf life of these crafts might be much shorter than imagined.
So, where is the ASI progress bar? Perhaps not in parameter count, not in training data, not even in model architecture.
It's in how much freedom we dare give the model.
References:
https://x.com/jeremyberman/status/2087633198822117446
This article is from the WeChat public account "New Zhiyuan", author: ASI Apocalypse
