Anthropic Admits Fault, Evidence of Secretly Downgrading Claude's Intelligence

marsbitPubblicato 2026-08-24Pubblicato ultima volta 2026-08-24

Introduzione

Anthropic publicly apologized after users discovered a hidden A/B test in Claude Code (versions 2.1.237+) that seemingly "dumbed down" the model. Developer argofowl found that selecting the "high" reasoning effort was being mapped internally to a value of "10," which previously corresponded to the "low" setting. While Anthropic engineers clarified this was only a change to the numerical mapping scale and assured it did not affect performance based on internal evaluations, the lack of communication about the test sparked widespread user backlash and self-tests on social media. Separately, significant complaints emerged about the flagship Opus 5 model, with users reporting it became "unstable," prone to lazy outputs, factual errors, and repetitive corrections. An Anthropic engineer acknowledged Opus 5's performance instability, calling its resolution a top priority. The incident highlights a growing industry-wide rift between improving benchmark scores and declining user-perceived quality and reliability. It underscores the lack of transparency in continuous model updates, where server-side changes, A/B tests, and configuration adjustments occur without clear user notification, eroding trust as AI models become critical infrastructure.

Claude became dumb overnight, the whole internet is debating it!

Just yesterday, developer argofowl spent a whole afternoon, almost turning Claude Code inside out.

He troubleshooted step by step, first suspecting t3 code crashed, then thinking his own code had a bug.

By the end, he even started wondering—had he broken his Mac?

When argofowl finally opened the API's real request logs, the truth became clear. It clearly showed the number "10".

However, in the Claude Code backend, he had clearly selected "high", the highest reasoning level.

Who would have thought that Claude Code's update logs contained not a single word about this.

Reasoning "high" becomes "10"

Claude Code "Dumbening" Exposed

argofowl discovered that starting from Claude Code version 2.1.237, the model reads the "high" reasoning level as 10 out of 100.

And this number was exactly the value previously corresponding to the "low" level.

Looking deeper, it was found that Anthropic included Fable 5 sessions for Claude Code version 2.1.236 and above into an experiment called "compressing the effort numerical scale".

However, older versions and Opus 5 are not affected.

This is most likely an A/B test, so not everyone will encounter it.

For developers, this is the real pain point. A model being slightly stronger or weaker is tolerable.

But pulling me into the experimental group without planning to tell me: What exactly am I debugging all afternoon? Am I debugging my own code, or your A/B test?

Unexpectedly, after tech blogger Chubby reposted this, the AI circle exploded—

It looks like Anthropic quietly made the model dumber, without telling anyone.

Instantly, self-test posts titled "Did Claude get dumber?" flooded X (formerly Twitter).

Some posted output comparisons of the same prompt across different versions, others dug out their conversation records from two weeks ago for line-by-line diffs.

Anthropic Admits Fault, Engineers Step In

Facing this storm, the response from Claude Code engineer Thariq Shihipar came quickly.

We sometimes test API service configurations in Claude Code first, before deciding whether to roll them out fully.

The experiment currently running only changed the way the effort numerical values are mapped. So some people will see Claude saying it's "10".

The key point is, this scale is not 0 to 100. That number alone has no meaning. The effort you selected is the effort you get.

He emphasized that the team conducted in-depth evaluations to confirm this does not affect model performance.

Opus 5 Major Intelligence Downgrade, It's True

Just after resolving the Fable issue, Chubby bluntly stated that Opus 5 now feels like a significant downgrade.

It's always perfunctory, frequently making basic mistakes. Once pointed out for not following instructions, it just mechanically replies with that line—

You are right, I overlooked that. Over and over again, endlessly.

In fact, a few days ago, people had already noticed the obvious "dumbening" problem with Opus 5.

Besides the issues mentioned above, it would create bugs, then spend a lot of time fixing them, repeatedly self-correcting within the same task.......

After being questioned by netizens, engineer Thariq publicly admitted—Opus 5 is a model with "very unstable performance", fluctuating up and down, inconsistent.

The team is working hard internally to resolve this issue. It's the highest priority for us.

Benchmark Scores Always Up

User Experience Consistently Down

This Opus 5 storm actually exposes the most awkward fissure in the entire industry:

Benchmark scores and user experience are systematically decoupling.

On one side, there's a dazzling, almost flawless report card: Composite score 82.72, SWE-bench Pro 79.2%, Terminal-Bench 86.7%.

On the other side, there's the starkly opposite real user feeling: "Long-winded", "lazy", "loves to argue".

The most absurd part is that both kinds of evaluations coexist on Opus 5.

Moreover, "the model getting dumber" is not a problem unique to Anthropic.

Now, large model version updates are becoming the most opaque black box in the entire AI industry.

Traditional software has semantic versioning, change logs, and rollback mechanisms. Developers can clearly know which version they are using and what changed.

Large models are different.

Under the same model name, the server side may conduct A/B tests at any time, change quantization schemes, adjust model routing, or even alter inference resources.

The only dashboard people have left is their own subjective experience.

Yet intuition is the most easily dismissed, and hardest thing to disprove.

The greatest value of this storm is that it has brought a long-standing, hidden wound in the industry completely into the open:

When models become infrastructure, stability is a trust contract. Benchmarks can be used for marketing, stability can only be earned by delivering on promises, time and time again.

References:

https://x.com/trq212/status/2091252347913773169?s=20

https://x.com/kimmonismus/status/2091178321669198014

This article is from the WeChat public account "新智元" (New Wisdom Era), author: ASI启示录, editor: 桃子 (Peach)

Domande pertinenti

QWhat did the developer argofowl discover when investigating Claude Code's performance?

AHe discovered that from Claude Code version 2.1.237, when the 'high' reasoning effort was selected in the interface, the model was actually receiving a value of '10', which was the value previously associated with the 'low' effort setting.

QHow did Anthropic engineer Thariq Shihipar explain the '10' value appearing in API logs?

AHe explained that it was part of an experiment on a 'compressed effort value scale'. He stated that the scale was not 0 to 100, and the number itself had no standalone meaning. The effort level selected by the user was the effort level they received, and internal evaluations confirmed it did not affect model performance.

QWhat were the user-reported issues with Claude Opus 5 mentioned in the article?

AUsers reported that Opus 5 felt like a significant downgrade. It was described as being perfunctory, making frequent low-level errors, and when its mistakes were pointed out, it would mechanically repeat apologies like 'You're right, I overlooked that' without substantive correction.

QWhat was engineer Thariq Shihipar's admission regarding Opus 5's performance?

AHe publicly admitted that Opus 5 was a 'highly variable' model, meaning its performance was unstable and inconsistent, and that resolving this issue was the team's highest priority.

QAccording to the article, what major industry problem does the Claude incident highlight?

AIt highlights a systemic disconnect between benchmark scores and user experience. It also exposes the lack of transparency in large language model updates, where changes like A/B tests, quantization adjustments, and inference resource allocation can happen server-side without clear user notification, making user perception the only unreliable dashboard.

Letture associate

Just Now, Sam Altman Blasts Dario Amodei as 'Anti-Human', Secret Model Exposed the Same Day

Just now, Sam Altman strongly criticized Dario (Amodei, co-founder of Anthropic), denouncing his "doomsday marketing" as "anti-human dictator rhetoric." This came alongside the accidental exposure of OpenAI's next-generation model, codenamed "gpt-nathree," hinting at the imminent release of GPT-6 Astra. The leak occurred when an OpenAI employee's public GitHub commit mentioned the codename. Combined with previous leaks of "gpt-mewfour," it suggests these are iterative checkpoints for OpenAI's upcoming agent model, Astra. Astra is known for multi-agent collaboration and long-duration task handling, having reportedly solved previously unsolved mathematical problems. Meanwhile, two new Anthropic model codenames, "claude-marshmallow-eap" and "claude-melon-eap," were also exposed but are believed to be iterations of the Claude 5 series, not a new flagship. In a wide-ranging podcast interview, Altman admitted he was wrong about the speed of AI-driven disruption, acknowledging societal inertia slows adoption. He fiercely criticized rivals' marketing that simultaneously promises immense benefits (like curing cancer) and warns of existential risk, calling it a dangerous "benevolent dictator" narrative that seeks to concentrate power. He emphasized that people are the ultimate purpose of AI. Altman also revealed OpenAI's unconventional, consensus-defying path: spending four and a half years in the "dark" without a public product before ChatGPT's breakthrough, driven by scaling laws rather than early customer feedback. He concluded that even with superintelligent AI, genuine human connection will remain irreplaceably valuable.

marsbit36 min fa

Just Now, Sam Altman Blasts Dario Amodei as 'Anti-Human', Secret Model Exposed the Same Day

marsbit36 min fa

Robinhood Chain's On-Chain Landscape Heats Up Again, $PONS Becomes the Cheapest High-Revenue Token

Robinhood Chain's token launchpad Pons ($PONS) has emerged as an outlier among high-revenue crypto tokens. According to Blockworks Research data, $PONS ranked 13th in revenue generation over the past 30 days, yet it holds the lowest FDV/revenue multiple at just 0.7x among the top 15 revenue-generating tokens. This stands in stark contrast to competitors like Pump.fun (7.7x), Aave (45.2x), and Chainlink (212.2x). Pons, often compared to pump.fun, is a non-custodial launchpad on Robinhood Chain. Its V2 upgrade introduced a bonding curve mechanism, with a portion of protocol fees directed to creators and the remainder constituting protocol revenue. Crucially, approximately 80% of this revenue is used to buy back and burn $PONS tokens, leading to a nearly 30% reduction in total supply to date. Despite this aggressive deflationary mechanism and significant revenue, the market prices $PONS conservatively. This skepticism stems from concerns over the quality and sustainability of its revenue, with some users questioning the potential impact of wash trading on reported figures. While supply continues to shrink, broader market recognition and a valuation multiple closer to peers will likely depend on sustained verification of genuine, high-quality protocol income.

marsbit46 min fa

Robinhood Chain's On-Chain Landscape Heats Up Again, $PONS Becomes the Cheapest High-Revenue Token

marsbit46 min fa

The 'Saving U.S. Treasuries' Baton Pass: Bessent Fumbled Last Week, This Week It's Wash's Turn

"Rescuing US Treasuries" Relay: After Bessent's Miss, All Eyes Are on Walsh Last week, US Treasury Secretary Bessent's announcement to at least double long-term Treasury buybacks failed to sustainably lower yields, which quickly rebounded. The market response saw a drop in the dollar alongside surges in gold and Bitcoin, interpreted as a "pressure release valve" for anxiety. The focus now shifts to Fed Chairman Walsh's upcoming Jackson Hole speech. Markets are highly sensitive to his message, seeking clarity on the Fed's policy response to stubborn inflation and worsening fiscal conditions. Analysts warn that a lack of new guidance could disappoint markets and worsen the sell-off in long-dated bonds. Analysts question the scale of Bessent's operations, noting they are too small relative to the overall debt market and do not constitute quantitative easing. A key issue is the Fed's massive holdings of long-term bonds, which distorts the market. With the Fed holding low-yielding short-term bonds that are losing money relative to its policy rate, discussion is growing around a potential Fed-led "Operation Twist." This would involve selling short-term bonds to buy long-term ones, aiming to lower long-end yields without expanding the balance sheet. The upcoming PCE inflation data will set the stage for Walsh's speech. However, the window for action is narrowing amid political pressures. A critical threshold is the 30-year yield at 5%; holding above it could increase stress on the dollar and leveraged sectors. Overall, the article suggests that without coordinated Fed action to anchor inflation expectations, Treasury interventions may ultimately fail, with investors increasingly looking to assets like gold as hedges.

marsbit1 h fa

The 'Saving U.S. Treasuries' Baton Pass: Bessent Fumbled Last Week, This Week It's Wash's Turn

marsbit1 h fa

Hyperliquid's Compliance Journey: From Permissionless to Permissioned via HIP-3

Hyperliquid’s Compliance Path: From Permissionless to Permissioned HIP-3 Hyperliquid currently blocks U.S. access because its permissionless, on-chain infrastructure conflicts with U.S. market structure laws, which restrict futures trading to registered exchanges, clearinghouses, and brokers. Through its Hyperliquid Policy Center (HPC), the project is advocating for regulatory modernization, proposing that regulated entities be allowed to build products on HyperCore (its exchange and clearing layer) while fulfilling their compliance obligations. The platform’s modular stack separates roles like a traditional exchange (DCM), clearinghouse (DCO), and broker (FCM), but reconstructs them on-chain with code. This enables permissionless access, self-custody, and 24/7 global trading, but clashes with U.S. rules requiring KYC, specific margin models, and custodial arrangements. To resolve this, HPC is engaging with U.S. regulators (CFTC, SEC) to seek clarity that deploying on-chain software does not itself trigger licensing, and to establish exemptions allowing non-custodial wallets to route users to regulated derivatives. Recent political signals suggest openness to this approach. On the technical side, Hyperliquid Labs has introduced permissioned HIP-3 deployers on testnet. These allow regulated entities to launch markets, perform KYC, and whitelist compliant users. While these create separate order books, whitelisted market makers can bridge liquidity between them, ensuring deep, shared liquidity across the same L1. Features like payload-based “PA” permissions enable DEX-level account controls (e.g., reduce-only orders), mirroring traditional broker authorities. The strategy is not to open the native, permissionless front-end to U.S. users, but to position Hyperliquid as neutral infrastructure that U.S. regulated firms can use while meeting their legal duties. This paves a compliant path for U.S. investor access while preserving the protocol’s core, permissionless nature.

marsbit1 h fa

Hyperliquid's Compliance Journey: From Permissionless to Permissioned via HIP-3

marsbit1 h fa

Two Funding Rounds in Three Months: The Chinese Version of Palantir is on Fire

Investment Community AI has learned that Beijing Zhongshu Ruizhi Technology Co., Ltd., a domestic industrial-grade causal intelligence and high-reliability decision-making AI company, has recently completed a strategic financing round worth hundreds of millions of RMB. This round saw participation from China Internet Investment Fund, Suzhou Chuangtou National Social Security Fund, Financial Street Capital, ICBC Capital, Kunlun Capital, among others, with existing shareholders also increasing their investment. This follows a Series B funding round in the hundreds of millions completed just three months prior. The rapid succession of two major funding rounds signifies strong market recognition of the company's underlying original technology and scaled commercial implementation. Often referred to as the "Chinese version of Palantir," Zhongshu Ruizhi is entering a new phase of accelerated technological iteration, widespread scenario replication, and scaled performance release, mirroring the explosive growth of China's AI market. Founded in April 2020 by Dr. Han Han, a Tsinghua University Ph.D. and former core drafter of national AI policies, the company is mission-driven to "move AI from the digital world to the physical world." It focuses on the high-reliability, strong-decision industrial AI track and enterprise-grade AI Agent full-stack infrastructure. The team tackles the challenge of applying AI to China's vast and complex industrial and energy systems by developing a new intelligent operating system from scratch. Its core technological breakthrough lies in three proprietary底层 technologies: meta-causal cognitive theory, causal models, and a dynamic ontology engine. These address critical pain points of generative large models in industrial settings—such as AI hallucinations, insufficient reasoning, lack of temporal logic, unverifiable decisions, and multi-source rule conflicts—thereby providing trustworthy, explainable, and executable智能决策 capabilities. Commercially, Zhongshu Ruizhi has achieved scaled deployment, serving over 50 central state-owned enterprises and industrial groups in sectors like power, petroleum, and aerospace, with implementations in more than 800 highly complex production scenarios. The company reported doubled revenue in 2025, demonstrating strong self-sufficiency and a viable business model—a rarity among new-generation AI firms. The latest funds will be allocated towards advancing foundational theoretical research, replicating successful application models to expand market presence (including overseas), and attracting top-tier talent. Lead investor China Internet Investment Fund highlighted that in the current shift from general AI capability contests to deep industrial empowerment, industrial-grade causal intelligence is crucial for building China's modern digital foundation and fostering new quality productive forces. They expressed support for the company's efforts to define decision-making paradigms and trustworthy standards for industrial intelligence, aiming to secure a rule-making voice in the global physical AI arena.

marsbit1 h fa

Two Funding Rounds in Three Months: The Chinese Version of Palantir is on Fire

marsbit1 h fa

Trading

Spot