Is the algorithm that trains all AI "sentenced to death" by AI itself?
Recently, two researchers from Tsinghua University and the Wharton School of the University of Pennsylvania posted a new paper giving a conclusion that the optimization theory community has been waiting 40 years for—
For gradient descent to reach maximum speed, merely tuning the step size is useless.
This is the first time in history that someone has proven the existence of a mathematical ceiling that cannot be surpassed for gradient descent by designing step size sequences alone.
And the one who completed the core proof was not a human, but GPT-5.6 Sol Pro.
GPT-5.6 Solved a Problem Unanswered for 40 Years
Here's what happened.
Everyone knows gradient descent; everything from GPT to Stable Diffusion to autonomous vehicles runs on it. The standard convergence rate for gradient descent is O(1/T); after T steps, the error roughly reduces to the order of 1/T.
In 1983, Nesterov added momentum to gradient descent, directly pushing it to O(1/T2). Running the same 1000 steps, the error goes from one-thousandth to one-millionth, a difference of three orders of magnitude. It remains theoretically optimal to this day.
A very natural question then arises: without adding momentum or changing the structure, can we purely rely on carefully designing the step size at each step to catch up with Nesterov?
This problem hung for a full 40 years. It wasn't until 2023 that MIT's Altschuler and Parrilo came up with the silver stepsize.
This set of step size sequences is not the traditional gradual decrease but fluctuates in size, showing a fractal self-similar structure. With it, gradient descent was pushed to O(T^{-1.2716}).
So, is this 1.2716 the final limit of pure step size scheduling, or just a starting point?
Recently, a pair of Chinese mentor and disciple tackled this problem.
Jianhao Ma just joined the Department of Industrial Engineering at Tsinghua University this July. He holds a Ph.D. from the University of Michigan and secured his faculty position after completing a postdoc at the University of Pennsylvania.
His postdoc supervisor, Yuxin Chen, is an endowed chair professor at the Wharton School, holds a Ph.D. from Stanford, moved from Princeton to UPenn, and has won the SIAM Best Paper Prize.
Previously, everyone was doing additions, designing smarter step size sequences to see how much the speed could be increased.
Ma and Chen's idea was to go the opposite way, proving the existence of a line that cannot be crossed no matter how the step sizes are designed.
To find a good set of step sizes, you only need one successful example. But to prove that "no possible step sizes work" is to say "no" to infinitely many possibilities.
The two pondered for a while, then directly threw the problem to GPT-5.6 Sol Pro, letting the AI give it a try.
Specifically, they gave GPT two things.
One was the research objective: to prove that pure step size scheduling cannot achieve O(1/T2). The other was a high-level strategy called the "resisting oracle."
Its principle is to first construct an adversarial trajectory that makes gradient descent walk the slowest, then find a real smooth convex function such that the path taken by gradient descent on this function is exactly this slow path.
After setting the direction, GPT-5.6 Sol Pro got to work.
The core solution it ultimately provided was a geometric construction.
Given any predetermined step size sequence, first pick out the "long steps," i.e., those steps where the step size exceeds the standard safe value of 1/L. Then place a set of mutually orthogonal anchor points in high-dimensional space, each corresponding to a long step.
Gradient descent is forced to walk along the same direction between two long steps, jumping to the next completely perpendicular direction upon encountering a long step. The entire trajectory is precisely implemented by a smooth convex function called the Moreau envelope, strictly equivalent.
The key to this construction is that it is tailor-made for your step size sequence. No matter how you design the step sizes, it can create a corresponding function to block you.
But the proof wasn't finished here.
The final lower bound cannot depend on the chronological order in which the long steps appear; otherwise, rearranging the same set of step sizes might allow escape.
GPT-5.6 found a matching technique, arranging the long steps by size, constructing a path, splitting them into odd and even groups for matching, completely eliminating the temporal dependency. Then it introduced a Lyapunov potential function to control global growth, combined with a truncation argument, aggregating local constraints into an overall lower bound.
This line of argumentation was formed only after Ma and Chen repeatedly interacted with GPT-5.6 Sol Pro, pointing out flaws in the derivation, having GPT correct them and continue, going through multiple iterations.
In Ma's own words, no non-trivial mathematical component in the core proof came from humans.
In the entire proof, there is a key parameter constrained by two conditions simultaneously: the matching bound gives a lower limit, and the growth control gives an upper limit.
As the convergence exponent p decreases, the two constraints tighten. At p = √(2+√3) ≈ 1.9319, the two lines meet, and the parameter's room for maneuver vanishes. Pushing further down, the proof becomes infeasible.
The conclusion finally given by GPT-5.6 Sol Pro is that for any predetermined non-negative step size sequence, the convergence rate lower bound for gradient descent is Ω(T^{-1.9319}).
Gradient descent relying purely on tuning step sizes, no matter how cleverly the sequence is designed, can never surpass this line.
In other words, to achieve the fastest convergence speed, the algorithm structure must be changed.
Lean 4 Final Review: Zero sorry, Zero admit
An AI-written proof, how to confirm it's not a hallucination?
Ma and Chen used the most hardcore verification method in the mathematical world: the Lean 4 theorem prover.
They used Codex to gradually translate GPT-5.6 Sol Pro's natural language proof into Lean 4 code.
This formal verification system checks every line of derivation; any logical leap or missing basis results in an immediate compilation error.
If a certain step truly cannot be proven, one can insert a "sorry" or "admit" to skip it temporarily—meaning "I haven't finished proving this step yet."
The final report card: Zero sorry, zero admit. Not a single step was skipped.
The code is public on GitHub, accompanied by a TRACEABILITY.md file that cross-references each theorem in the paper with its corresponding proof in the Lean code line by line. Anyone who wants to check can compile it themselves.
Project address: https://github.com/jianhaoma/gd-lower-bound-lean
The entire verification chain is a three-part relay. GPT-5.6 Sol Pro constructs the proof, Codex translates it into Lean 4, and the compiler performs the line-by-line final review. Humans supervise the entire process.
You don't need to "believe" the AI; let the formal system be the judge.
The Story Isn't Over
The currently confirmed range is this: silver stepsize has pushed gradient descent to T^{-1.2716}, and Ma and Chen have proven it cannot exceed T^{-1.9319}.
There's still a gap of 0.66 in between. Where is the true limit?
Ben Grimmer, an optimization scholar who has long studied this problem, stated after reading the paper that he "strongly believes" 1.2716 is the true ceiling.
If he is correct, then silver stepsize is already the end of pure step size scheduling, and there is room to further tighten the lower bound provided by Ma and Chen.
But regardless of where the true limit lies, this paper has already completed the most crucial step: Relying purely on tuning step sizes, gradient descent cannot achieve a perfect score. This has gone from speculation to theorem.
And this result was achieved by just two people. No mathematics team, no Lean experts, no dedicated computing budget, using the commercially available GPT-5.6 Sol Pro that anyone can access.
If this model can be replicated, any researcher in the world with a good problem could have the AI run the proof for them.
References:
https://arxiv.org/abs/2608.10418
This article is from the WeChat public account "New Zhiyuan," author: ASI Revelation, editor: Moses
