AI-powered scrutiny of top scientific journals reveals that 99.2% of papers contain at least one error, challenging the perception of robust peer review. A recent study using AI agents to audit papers from ICML 2026 found that 58 out of 92 reviewed papers could not be fully reproduced. Reasons for failed replication include missing code, broken dependencies, and results inconsistent with claims. Separately, a GPT-5-based checker analyzed established AI conference papers, detecting an average of 4.7 objective errors per paper, with mathematical mistakes being most common. This trend suggests a growing "reproducibility crisis" as paper volume and complexity outpace traditional verification. However, it also presents an opportunity: researchers can now use AI to efficiently audit past literature, identify errors in foundational work, and publish corrections—a potentially fruitful new research avenue. In one case, AI even corrected century-old chemical data that had been accepted as fact. While AI tools significantly lower the cost of verification, they are not infallible (e.g., 83.2% precision rate in one system) and human oversight remains crucial. The era of AI-assisted verification may redefine the scientific process, where publication marks not an end, but the beginning of automated validation.
marsbit5天前




