Who still says there are no new problems left in scientific research? That all the good directions have been taken by predecessors???
What if I told you—
Most of the papers already published in top journals have issues?
Recently, researchers using AI Agents for "academic fact-checking" discovered:
Among 92 papers reported at the 2026 International Conference on Machine Learning (ICML), 58 could no longer be reproduced.
Really?? Could it be that the opportunity to publish heavily in top journals is actually here?
Top Conference Papers Are Being Collectively "Fact-Checked" by AI
Typically, a paper accepted into a top-tier conference means it has undergone peer review.
However, limited by time, peer reviewers hardly have time to download data from scratch, run models, or reproduce experiments to verify every experimental conclusion in the paper.
Meanwhile, with the explosive growth in AI research papers, models becoming increasingly complex, and experiments growing larger, the code, data, and parameter settings behind a single paper may involve hundreds of details.
The verifiability of papers has thus become increasingly important. A paper is published, but can its conclusions be re-run?
Currently, AI Agents have significantly reduced the cost of such verification and reproduction.
On July 22, a US research auditing company used AI Agents to conduct a systematic result reproducibility audit on all 168 orally presented papers at ICML 2026.
Among the 168 papers, 92 contained at least 5 verifiable conclusive statements.
The final result: only 34 papers had over 40% of their conclusions successfully reproduced by the AI Agent; only 8 papers had over 80% of their conclusions reproduced.
However, it must be clarified that "failure to reproduce" is vastly different from "research fraud."
Reasons for reproduction failure include, but are not limited to: code missing key files, broken dependency library versions, results not matching the paper, and 4 papers whose dependent models had been taken offline, meaning the experimental results could permanently no longer be reproduced by anyone.
Besides these, some errors were quite egregious—
For example, one paper touted "training only 0.77% of the base model's parameters" as its core selling point, but its actually open-sourced checkpoint trained 6.31% of the parameters, a difference of approximately 8 times.
Another paper included a reliability table based on a certain evaluation model, but its open-source code did not contain that evaluation model at all, nor any script capable of generating the results in the table.
It's not an isolated case. In 2026, Hugging Face and AlphaXiv jointly launched the 「Agent Reproduction Challenge」 competition, inviting researchers to use AI Coding Agents to automatically reproduce papers accepted at ICML 2026, with similarly unoptimistic results.
A similar trend is even more striking in the detection of paper errors and omissions.
At the end of 2025, a study developed a GPT-5-based paper inspection system to analyze papers already published in top AI conferences and journals, searching for objectively verifiable problems. The researchers explicitly stated:
We only look for "objective errors"; we do not judge a paper's novelty or research value.
The final results showed an average of 4.7 objective errors detected per paper, with 99.2% of papers flagged for at least one problem.
△ Image generated by AI
Regarding error types, mathematics and formula errors accounted for the highest proportion at 54.0% (including incorrect equations, logical flaws in derivations, incorrect assumptions in proofs, etc.).
Approximately 30.8% of NeurIPS papers and 23.8% of ICLR papers contained at least one substantive error that could affect result interpretation.
More notably, the time trend: the average number of errors per NeurIPS paper increased from 3.8 in 2021 to 5.9 in 2025, a rise of 55.3%.
......
Perhaps in the future, publishing a paper will no longer signify the "victory" of research but merely mark the beginning of another round of AI verification.
A Major Boon for Academic Newcomers: Publishing in Top Journals Without Conducting Experiments
So, friends, for those worried about choosing a research topic, this might actually be a rather unexpected "benefit."
Searching literature for errors and writing errata has always been a legitimate form of academic output. Now it seems like a completely untapped academic blue ocean.
How to get started specifically? Here are a few tips for academic newcomers struggling with topic selection:
First, shift research thinking: don't compete at the frontier, compete with the past.
Move from "finding new topics" to "finding anomalies in old papers," focusing especially on classic papers that are highly cited but have little controversy. After all, "99.2% of papers have at least one error."
Second, let AI help you create knowledge maps and research lineages.
This kind of knowledge map can help you understand—which are the truly foundational works? Which research viewpoints currently have controversy? Which conclusions are heavily cited but lack verification? What are the citation relationships between different papers? From this, you can find connections and anomalies that were difficult to discover in the past.
Third, Ask anything.
Many important breakthroughs in the history of science did not come from posing entirely new questions but from re-examining a long-accepted assumption.
For example: Has a classic experiment been verified by today's standards? Does a widely cited conclusion have unnoticed limitations or conditions?
In the past, this kind of inquiry was very costly, requiring researchers to spend a great deal of time consulting original literature and comparing experimental details. Now, AI has reduced this cost to almost zero.
AI May Begin to Reorganize the History of Science
Recently, theoretical chemist Pios from a Zhejiang laboratory, while using AI to predict molecular boiling points, found that the results clashed significantly with a chemical database from 75 years ago.
Anyone's first reaction to this might be, "Well, I must be wrong then."
Pios was no exception; he quickly checked his model. But after manually tracing back to the original literature, he discovered: Goodness, it was the AI that was actually correct.
Greatly surprised, Pios continued using AI to check further and even found that a boiling point measurement from about a century ago, considered authoritative by the academic community, was also wrong.
It's important to note that this data had been cited and absorbed into subsequent research over many years.
The fact that such problems remained undiscovered for so long might be directly related to the sheer scale of scientific literature.
△ Image generated by AI
The volume of modern scientific papers has far exceeded the reading capacity of any individual researcher. For example, the annual submission volume for just one top AI conference, ICLR, rose from 1013 in 2018 to 19619 in 2026.
At this scale, an initial error appearing in a paper, once repeatedly cited by subsequent literature, continues to propagate along the citation chain, forming a de facto academic consensus.
Due to their age, complex citation chains, the enormous time required for verification, and limited academic reward, the vast majority of errors that have entered the literature system have never been systematically reviewed.
But now everything is loosening up. At least in terms of technical capability, people possess for the first time the possibility of large-scale re-examination of past scientific literature.
However, taking the GPT-5-powered Paper Correctness Checker as an example, its detection precision rate is 83.2%, and in each detection, about 40% of real errors still remain undetected.
It can be said that such AI fact-checking tools alone are insufficient to act as judges of scientific literature. The output results of AI verification tools ultimately still require human review.
This article is from WeChat public account "QbitAI" (ID: QbitAI), author: Cheng Qian
