Benchmarking has felt a bit off lately.
On September 8th, the analysis firm SemiAnalysis fired off five tweets, directly naming Google and Meta, two trillion-dollar giants—
Calling Gemini 3.8 Flash and Muse Spark 1.3 "the two models with the most obvious signs of benchmark gaming."
The evidence is crystal clear.
On the Terminal-Bench 2.1 leaderboard, which tests Agent task-completion capabilities, Gemini 3.8 Flash scored 89.4 points, ranking 2nd out of 182 models, placing it ahead of GPT-6 Astra.
But then, on the newly launched Terminal-Bench 4.0 on August 29th, the same model scored only 19.1 points, dropping directly to 12th place out of 14 tested, its score instantly slashed by 80%.
During the same period, GPT-6 Astra dropped from 88.4 to 57.7, a more reasonable ~35% drop.
Just 27 minutes later, Meta's Chief AI Officer, Alexandr Wang, stormed into the comments and opened with six words: This is a dumb argument.
He pointed out that GPT-5.6 Sol scored 88.8 on v2.1 but dropped to 37.3 on v4.0—an even bigger discrepancy—yet no one accused Sol of gaming the benchmark.
He also emphasized that Meta never claimed Muse Spark 1.3 was as powerful as Astra or Fable 5.1, just that its cost-performance ratio is obviously better.
A New Test Paper, Scores Plummet
To briefly explain, Terminal-Bench measures an Agent's real-world task-completion ability.
The method involves giving the model a terminal and a vague goal, then letting it figure out the rest on its own. The system must then plan its own path, call tools, write scripts, and even fix its own errors.
On the v2.1 version, which has been gamed by the industry for half a year, these two models' scores did look impressive.
Gemini 3.8 Flash scored 89.4, ranking 2nd. Muse Spark 1.3 scored 88.8, ranking 4th, closely followed by GPT-6 Astra with 88.4 and Claude Fable 5.1 with 85.02.
Wow. A cheap Flash-tier model and Meta's new cost-effective offering, both rubbing the flagships of the two top labs into the dust.
Then, version 4.0 arrived.
The new test paper has 66 questions. It not only removed 8 old questions that had been "solved," significantly revised 19, but also uniformly imposed an 8-hour time limit.
Anti-cheating mechanisms were also ramped up to extreme levels.
The organizers not only set 35 detailed scoring rules but also added a dedicated "adversarial cheating test," intentionally letting the Agent try to exploit loopholes in the reward mechanism. Any question that could be exploited was immediately invalidated.
Once the new leaderboard was out, the atmosphere changed completely.
Claude Mythos 5.1(max) held steady with 60.9 points. GPT-6 Astra and Claude Fable 5.1 followed with 57.7 and 55.8 points respectively, then Claude Opus 5 with 52.3. The previous generation's GPT-5.6 Sol and Terra scored 37.3 and 23.6 respectively.
In contrast, Gemini 3.8 Flash plummeted to 19.1 points. And according to the chart provided by SemiAnalysis, Muse Spark 1.3 scored 33.3 points.
To summarize.
On the old leaderboard, Gemini 3.8 Flash could still beat GPT-6 Astra. On the new one, its score is less than a third of Astra's, even lagging behind the previous generation's GPT-5.6 Terra.
No Need to Copy Answers, Just Buy an "Exam Cram"
Arguments aside, SemiAnalysis's second tweet's accusation reveals the real inside story of the industry.
In their view, the tasks for Terminal Bench 2.1 were completely public. Meta and Google absolutely wouldn't be stupid enough to train directly on the original questions, but they would definitely buy data—specifically training data designed to be infinitely close to TB 2.1 question types. The final effect is practically no different from cheating.
And this is the high-level play for gaming benchmarks in 2026.
The old "contamination" method was crude: directly mixing original questions into pre-training corpora for the model to memorize.
This was easy to catch. A single data wash would expose the true performance.
The industry has stumbled on this many times. For example, nearly 40% of HumanEval samples were contaminated; GSM8K scores dropped by 13 points after decontamination. The most severe was SWE-bench: retesting with a private code library caused scores to be halved.
So now, big companies don't touch original questions anymore. Instead, they buy "simulated exam papers that look like the real test." That is, closed task systems that provide the model with opportunities for trial and error and give rewards.
Currently, this is a mature business with clear price tags.
A single training task sells for $200 to $2,000. Complex software engineering tasks can even fetch $20,000.
If you want to clone a website to create a UI training ground, it costs about $20,000.
If you require replicating a product on the scale of Slack, that starts at $300,000.
For clients demanding exclusive buyouts, the price can quadruple or quintuple.
According to Epoch AI's research, Anthropic internally discussed spending $1 billion annually on this. Single-quarter contracts often run into six or seven figures, with some researchers revealing an average of $300,000 to $500,000 per quarter.
On the supply side, at least 35 companies are in this business, mostly small teams of under 20 people. Established data giants are also pivoting. Scale AI's revenue exceeded $1.4 billion before Meta's investment, and Surge's ARR is approaching $1 billion.
The current situation is utterly bizarre.
On one side, there are completely public leaderboard question banks. On the other, there's a mature industry selling test questions. To get a model to score high on a specific leaderboard, you really don't need to bother cheating; you can just pay for it.
Acting as Both Test-Seller and Proctor
Subsequently, SemiAnalysis directly named a company called Datacurve.
On the DeepSWE 1.1 leaderboard, which specializes in testing long-cycle programming, Muse Spark 1.3(max) took first place with 75.4 points. GPT-6 Astra and Claude Opus 5 followed closely, and Gemini 3.8 Flash scored 73.8, ranking fourth.
The two accused of benchmark gaming: one first, one fourth.
Presumably, you've already spotted the trick behind this.
This leaderboard is run by Datacurve. And Datacurve's money-making avenue happens to be selling "expert-level coding data and reinforcement learning environments" to frontier labs.
DeepSWE is not only Datacurve's own benchmark, but the scoring also uses its own evaluation environment.
Acting as both the tutoring agency selling test papers and the exam-setting proctor—this is practically confirmed.
The Shelf Life of Leaderboards is Running Out
Finally, SemiAnalysis delivered a verdict on the entire industry's public evaluation.
They believe this is ultimately the fate of all high-quality public benchmarks. TB 4.0 is no exception; it only appears accurate now because it's only been out for two weeks.
As long as its questions remain public, it won't be long before all labs claiming to be "frontier" wear them down to a polished sheen.
The antidote SemiAnalysis ultimately offers is very direct: create more high-quality private benchmarks.
The direction is right, but the cost is equally painful.
Once evaluations are fully privatized, public leaderboards will completely devolve into marketing materials for each company.
Truly differentiating real-world performance will only flow in dark channels between labs and private evaluation agencies.
Big companies will of course welcome this, as no one can memorize answers in advance for a closed-book exam.
But for the vast majority of developers, that public ruler used to fairly measure all models will likely never be found again.
References:
https://x.com/SemiAnalysis_/status/2097112791471522292
Editor: Moses
This article is from the WeChat public account "New Zhiyuan", author: ASI Apocalypse
