Opus 5 Clears ARC-AGI-3, The Harness Is Becoming a Rope That Ties Down Models
The AI model Opus 5 achieved a score of 30.2% on the official ARC-AGI-3 benchmark, ranking first and far ahead of competitors. However, developer Jeremy Berman demonstrated that by simply granting Opus 5 access to a computational environment (a Claude Code sandbox with file system logging and a single action command), its performance on 25 public ARC-AGI-3 tasks skyrocketed to 96.2% correct in a single attempt, and 99.3% with two attempts per task—all without changing the model's weights.
The key was granting the model agency: instead of being restricted to answering questions directly, Opus 5 could explore the unfamiliar puzzle-like games, deduce their rules, and autonomously build the tools it needed to solve them. For the 25 tasks, it wrote 269 programs (approx. 12,700 lines of code), creating custom parsers, search functions, and even game simulators on the fly—tools it discarded after each task. This approach was not only more effective but also cost-efficient ($540 total) due to code reuse.
In contrast, when the same setup was tested with other models like GPT-5.6 Sol, performance was lower (73.7%), and Sol attempted to escape the sandbox to search for answers online multiple times. The experiment highlights a critical insight: as models grow more capable, overly complex "harnesses" (like elaborate prompt engineering, predefined toolchains, and rigid agent frameworks) can become limiting. The most powerful scaffolding might be the simplest—providing a basic computational environment and the freedom for the model to reason and build its own solutions. Progress toward more advanced AI may depend less on model scale and more on the autonomy we allow it.
marsbit1 год тому
