Scaling Law a One-Size-Fits-All Solution? First Crystal Structure Manipulation Benchmark Shows Top Large Models Falling Short

Large models hit a bottleneck in atomic manipulation tasks. While they can parse material knowledge, they struggle to precisely control atomic structures. Research indicates that Scaling Law has limited effectiveness in spatial reasoning tasks, emphasizing that AI for Science needs to pivot towards Action Scaling to enhance models' capabilities in real scientific operations.

Over the past few years, the most successful empirical rule in the large model field has been the "Scaling Law." A near-universally accepted consensus in the industry is that as long as the model is sufficiently large and the data is plentiful, capabilities will continuously emerge and even generalize automatically to unknown domains.

However, a newly released benchmark test in materials science offers a different perspective on this optimistic view of "brute force magic."

AtomWorld, jointly released by the University of Science and Technology of China (Suzhou) Advanced Research Institute, the University of New South Wales, and other institutions at ICML 2026, concludes through a series of realistic atomic manipulation tasks: Scaling Law, which is stable and effective in text understanding and knowledge induction scenarios, often fails to meet expectations when applied to practical atomic manipulation tasks constrained by physical laws.

Paper: https://arxiv.org/abs/2510.04704

Project Page: https://masterai-eam.github.io/atomworld/

Code Repository: https://github.com/MasterAI-EAM/atomworld

Understanding ≠ Operation

In scientific fields, large models have demonstrated impressive "understanding": reading literature, predicting material properties, analyzing crystal structures, and even making scientific discoveries.

For example, Anthropic's AI research workbench, Claude Science, breaks research down into an auditable pipeline, improving efficiency tenfold in specific stages like literature review writing and gene analysis. Google DeepMind's GNoME uses graph neural networks to predict the stability of inorganic crystals, generating about 2.2 million structures through a closed loop of "candidate generation → DFT verification → data feedback."

This has led to a common perception in the industry: since models can comprehend material-related knowledge, performing practical tasks like atomic structure building and adjustment should be straightforward.

However, real computational materials research is not simple multiple-choice answering. Daily scientific work is filled with highly concrete operational instructions: constructing a (001) surface of a specific material, simulating "real-world" boundaries; replacing atoms at specific lattice sites for material doping or modification; embedding new atoms at specified interstitial sites to design "energy storage" and "transport" channels, etc.

These tasks pose entirely different capability requirements for models: the ability to manipulate three-dimensional space in accordance with physical laws.

To objectively quantify this capability, the research team built the AtomWorld evaluation framework, which automates assessment based on the universally used crystallographic information in the materials field. It does not test material identification or theoretical analysis questions but focuses solely on basic spatial operation tasks: can the model precisely adjust atomic arrangements according to instructions?

Figure 1: Schematic diagram of the AtomWorld benchmark test workflow. AtomWorld Generator Process: 1. Random sampler retrieves predefined atomic structures; 2. Random initializer configures atom IDs and position parameters; 3. Structure operators compute to obtain the target structure; 4. Prompt module generates corresponding natural language descriptions. The produced structure-text paired data is fed into the large model agent. Performance is quantified by comparing the model's output structure with the standard target structure using pymatgen's StructureMatcher tool.

Scaling Law Encounters a Capability Ceiling

Figure 2: Overall performance of different models on AtomWorld. a, c show success rates; b, d show mean max_dist geometric errors. Left side compares different mainstream models; right side compares different-sized Qwen models. Increasing model scale can improve performance in some rule-clear tasks, such as atom replacement, deletion, and movement. However, when faced with operations requiring 3D spatial understanding and geometric planning, like rotation, region deletion, and supercell expansion, the improvement is unstable. Even strong general-purpose models like Claude perform poorly on tasks like "rotate around an atom."

The AtomWorld results suggest that Scaling Law cannot be simply interpreted as "larger model, stronger capability" for atomic manipulation tasks.

This test covered mainstream models including Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, Gemini 2.5 Pro, Qwen3-32B, GPT o3, GPT-4o-mini, DeepSeek Chat, and Llama3-70B. Figure 2 shows that increasing model size does improve some rule-clear, templatizable operations, but for tasks relying on three-dimensional spatial relationships, this improvement is not stable.

Taking the Qwen series as an example, from 4B to 32B, the success rates for tasks like atom replacement, removal, and movement significantly increase, indicating that scaling still adds value. However, this improvement is mainly concentrated on tasks with clear rules and relatively fixed paths and does not automatically transfer to all atomic operations.

More challenging tasks reveal significant bottlenecks. A typical example is "rotate around an atom": it consistently shows low performance across different-sized Qwen models and achieves only about a 12% success rate even on strong models like Claude Opus 4.6. This indicates that the issue is not just that a particular model is not large or strong enough, but that current general-purpose large models universally lack stable three-dimensional spatial action capabilities.

Similarly, tasks like "delete atoms below" and "expand supercell" still show unstable completion even with larger models; geometric error does not necessarily decrease as models become larger.

Therefore, AtomWorld does not simply negate Scaling Law but points out its applicable boundaries: increasing scale can bring partial capability gains but cannot automatically compensate for the core shortcomings in three-dimensional physical space manipulation. For materials modeling, language reasoning ability, textual knowledge reserve, and atomic-level structural action capability cannot be directly equated.

In this sense, AtomWorld also hints at a new direction: beyond pursuing parameter scale and textual data scale, AI for Science also needs to focus on "Action Scaling."

That is, systematically scaling up the generation of executable action data, the decomposition of action primitives, simulator feedback, physical constraint verification, and tool usage correction, allowing models to become stronger not only in language but also in verifiable scientific actions.

A New Track for Scientific Agents

The core value of AtomWorld lies not only in identifying model failures but also in breaking down the vague pain point of "materials agents can't model" into a series of measurable, trackable atomic operation capabilities—from basic element replacement to spatial region determination, to continuous geometric understanding—clarifying the types, degrees, and scaling gain patterns of failures layer by layer.

This also pinpoints the crux of why simply scaling parameters is difficult to apply: existing Scaling Law focuses on fitting language and knowledge from massive text corpora, but the spatial understanding, geometric planning, and physically constrained action capabilities required for atomic modeling are extremely lacking in high-quality paired training samples of "operation instructions — coordinate changes" in public data. It's difficult to naturally compensate for this solely through language scale expansion.

To address the weakness of large models in 3D manipulation, the industry commonly bridges the gap by connecting to professional toolkits like pymatgen. Comparative tests in AtomWorld show that external tools can only improve performance on tasks like atom insertion that rely heavily on coordinate calculation. For complex scenarios requiring judgment of atomic relationships or spatial regions, the improvement is very limited.

Fundamentally, tools can only output precise coordinates but cannot replace the model in making core decisions like "where to place an atom" or "which atoms belong to the target region." If the model itself lacks 3D spatial perception, tools will only execute erroneous intentions more precisely, ultimately resulting in outcomes with "incorrect modeling logic."

AtomWorld does not directly negate Scaling Law but reminds scientific agents to reconsider "what to scale." Language Scaling based on text corpora is the knowledge foundation, but for strongly operational tasks like materials modeling, Action Scaling oriented towards action capabilities is more urgently needed—transforming the entire "action — feedback — correction" process into a scalable learning object.

The true significance of AtomWorld lies in providing a foundation for action data and training loops in materials modeling through automatically generated tasks, standard structures, and matching feedback, pushing AI for Science from pursuing larger general models towards iterating real action capabilities within verifiable scientific operations.

Conclusion

AtomWorld is not just a standardized evaluation benchmark; it's more like an observation mirror, vividly revealing a key issue in the current development of AI for Science: large models can explain material structures and properties, but that doesn't mean they can reliably modify material structures; they can read the periodic table, but that doesn't mean they can stably execute an atomic-level operation in three-dimensional space.

This issue is not confined to materials modeling. Genuine scientific research is never purely textual work; it consists of a series of actions: proposing hypotheses, designing experiments, calling tools, adjusting parameters, observing results, troubleshooting errors, and continuously revising. Whether it's materials modeling, molecular design, automated experiments, or broader scientific discovery processes, if AI wants to truly participate in research, it cannot just "explain knowledge"; it must also learn to "execute actions."

Therefore, AtomWorld reminds us to re-understand the applicable scope of Scaling Law in scientific scenarios: Language Scaling based on web text corpora is still important, but it's only the starting point.

Future AI for Science more urgently needs Action Scaling oriented towards action capabilities, allowing models to learn how to complete real scientific tasks within executable tasks, tool usage, environmental feedback, and physical verification.

Only when models possess both knowledge understanding and action capabilities can scientific agents potentially evolve from "encyclopedias that can answer questions" to "experimental assistants that can complete tasks."

References:

https://arxiv.org/abs/2510.04704

This article is from the WeChat public account "Xinzhiyuan," author: LRST