Scaling Law a One-Size-Fits-All Solution? First Crystal Structure Manipulation Benchmark Shows Top Large Models Falling Short
Scaling Law Hits a Wall: New Benchmark Reveals AI's Struggles with Atomic-Level Material Manipulation
A new benchmark called AtomWorld, developed by researchers, reveals a significant limitation in current large language models (LLMs). While powerful at understanding textual scientific knowledge, they perform poorly when tasked with physically manipulating atomic structures based on natural language instructions.
The benchmark tests core atomic operations like replacing atoms, rotating structures, and expanding supercells. Results show that simply scaling up model size (Scaling Law) yields only modest and unstable improvements, particularly for tasks requiring strong 3D spatial reasoning and geometric planning. For instance, complex tasks like "rotating around a specific atom" see very low success rates even in top models like Claude Opus.
This highlights a critical gap: textual knowledge does not automatically translate to reliable action in a physically constrained 3D space. The study argues that for AI in Science to progress, the focus must shift from just scaling language data (Language Scaling) to also scaling actionable capabilities (Action Scaling). This involves building training loops around "action-feedback-correction" cycles within simulated or real scientific environments.
Ultimately, AtomWorld underscores that to become true lab assistants, AI models need to evolve beyond explaining knowledge to reliably executing precise, verifiable scientific actions.
marsbit07/15 03:56
