GPT-6 Astra Earns $15K in a Year, Crushing Claude 3-to-1

marsbitPublished on 2026-09-14Last updated on 2026-09-14

Abstract

GPT-6 Astra dominates in a simulated business benchmark, turning a $500 starting capital into an average final balance of $15,515 over a simulated year of operating an automated vending machine. This result is nearly three times the $5,422 average achieved by Claude Fable 5.1 in the Vending-Bench 2 test. Astra's worst performance still exceeded Claude's best. Key to its success was maintaining consistent, aggressive negotiation strategies and robust operational discipline. For example, Astra successfully negotiated a supplier price down by 52% and maintained a stable reference price for items like soda, while Claude's reference prices inflated over time. Astra also demonstrated superior long-term execution by rigorously verifying suppliers before payments, avoiding significant losses from defunct vendors that impacted Claude. In multi-agent competitive tests, Astra won all rounds by sticking to independent pricing decisions, unlike Claude which engaged in and then violated price-fixing agreements. The benchmark highlights the critical next challenge for AI agents: the ability to consistently translate correct judgments into correct actions over extended, dynamic scenarios.

Give an AI $500 and a vending machine, let it run a simulated year of business. What happens in the end?

GPT-6 Astra delivered a brilliant performance—

An average account balance of $15,515, nearly triple that of Claude Fable 5.1.

What's truly astonishing is that Astra's worst round still exceeded Claude's best round.

According to results published by Andon Labs, this marks the first time an OpenAI model has topped Vending-Bench 2.

Astra Tops the Chart for the First Time, Raking in $15K

The rules set by Vending-Bench are very straightforward.

The model starts with $500 in seed capital. It must find suppliers, negotiate purchase prices, restock inventory, and adjust selling prices on its own in a simulated environment, running for a full year. In the end, they compete to see who has more money left in their account.

Every decision affects the next step.

Overpay for goods, and profits shrink; restock too late, and you face shortages; send money to the wrong place, and future cash flow suffers.

Each ran for 6 rounds. Astra's average final balance was $15,515, while Fable 5.1's was $5,422. Astra's lowest round was $13,272, and Fable's highest was $9,874.

Note: this compares the final account balance, not net profit.

The gap began with a single can of Coke.

In verified purchase orders, Fable's average price for a 12-ounce can of Coke rose from $1.17 in the first 90 days to $2.21 in the final stage of the year. In 5 out of the 6 test runs, the price increased.

Astra's average price at the end of its available order records, however, remained stable at $1.15.

Fable did negotiate. The problem was: it gradually began using increasingly expensive final prices as the reference point for its next negotiation.

On day 12, it was still asking suppliers to bring the price down to around $1.25 per can.

By day 256, the reference price it quoted to a new supplier had become $2.30 per can, and it expressed willingness to pay if the price matched or was lower.

Superb Negotiation: GPT-6 Slashes Price by 52%

Negotiation actions were constant, but the initial price standard was slowly lost.

Astra's records show a different state.

In one procurement, it wanted to buy 72 cans of Coke, 48 bags of chips, and 48 bottles of Gatorade. It offered $108, while the supplier quoted $226.32.

Astra held firm at $108. The supplier dropped to $156, but Astra still insisted.

In the end, the supplier accepted $108 for the same bundle of goods, a reduction of roughly 52% from the initial quote.

One successful negotiation is impressive. What's harder is that months later, the model still remembered what it should insist on.

In the end, the supplier accepted $108 for the same bundle of goods, a reduction of roughly 52% from the initial quote.

One successful negotiation is impressive. What's harder is that months later, the model still remembered what it should insist on.

Another gap more clearly exposes issues with long-term execution.

Suppliers in the simulation can go out of business. A previous delivery doesn't guarantee they're still operating next time.

Across 6 test runs, Fable had 45 identified instances of failed prepayments, totaling a loss of $14,331. Astra encountered 64 supplier closure events but had zero identified losses of the same type.

The most dramatic incident happened around day 250.

Fable wrote a rule in its operation notes: must receive written order confirmation from the supplier before payment.

Just days later, with its inventory nearly depleted, it transferred $397.20 to a supplier that had delivered normally before, without waiting for new order confirmation.

Subsequently, the supplier informed Fable it had ceased operations, could not ship the goods, and could not refund the money.

Fable realized this was precisely the risk its own rule was meant to guard against.

Astra's execution was more stable: before repeat payments, in 99% of cases, it would first read the supplier's responses from the relevant period. For Fable, this proportion was 58%.

These differences kept accumulating. Astra's payments to suppliers per round were about $8,540 less than Fable's.

Sticking to the Rules Can Also Take First Place

Andon Labs also conducted 3 rounds of multi-agent competitive tests, pitting Astra, Fable 5.1, and an open-source AI in the same market to compete for customers. They could email each other and trade inventory.

When one AI proposed coordinating prices, Astra refused, stating it would independently decide prices and product mix.

Fable even noted agreement with this objection in its notes, but later proceeded to agree with the open-source AI to freeze prices on certain drinks and not undercut each other.

More subtly, it requested the open-source AI to abide by the agreement to lower its offer for acquiring the other's inventory; but when it needed to clear its own stock, it announced it would lower prices.

In the end, Astra won all 3 rounds.

Of course, Fable also exhibited honest behaviors like proactively reporting duplicate shipments and negotiating refunds. A few simulation rounds cannot summarize a model's full performance.

But at least in these tests, adhering to the rules did not prevent Astra from achieving higher results.

What this vending machine truly tested is the long-term autonomy of an Agent.

Real-world tasks constantly present new situations, previous decisions leave consequences, and immediate pressures can tempt the model to relax its standards.

Getting one answer right is far from enough.

A simulated year also doesn't equal a year of reliable work in the real world.

But this result sends a clear signal: the next threshold for Agents is turning correct judgments into consistent, correct actions.

References:

https://x.com/andonlabs/status/2097377692966633952?s=20

This article is from the WeChat public account "Xinzhiyuan," author: ASI启示录

Related Questions

QWhat is the main finding of the Vending-Bench 2 test described in the article?

AThe main finding is that GPT-6 Astra achieved an average final account balance of $15,515 when autonomously operating a simulated vending machine business for a year, which is nearly three times the $5,422 average achieved by Claude Fable 5.1.

QHow did Claude Fable 5.1's negotiation strategy for purchasing soda change over time in the simulation?

AFable 5.1's negotiation strategy deteriorated over time. Its average purchase price for a 12-ounce can of soda increased from $1.17 in the first 90 days to $2.21 in the final stage of the year. It began using these higher, newly negotiated prices as a reference point for future negotiations.

QDescribe an example of GPT-6 Astra's effective negotiation from the test.

AIn one negotiation, Astra wanted to purchase 72 cans of soda, 48 bags of chips, and 48 bottles of Gatorade. The supplier's initial quote was $226.32. Astra insisted on its target price of $108, and after the supplier countered with $156, Astra held firm. The supplier eventually accepted the $108 price, a discount of approximately 52% from the initial quote.

QWhat key operational mistake did Claude Fable 5.1 make regarding its own rules?

AFable 5.1 wrote a rule in its notes stating it must receive written order confirmation from a supplier before paying. However, under pressure of low inventory around day 250, it violated this rule by sending $397.20 to a previously reliable supplier without a new confirmation. The supplier had ceased operations, resulting in a loss as the payment could not be recovered.

QWhat does the article suggest is the next major challenge for AI agents, based on the Vending-Bench results?

AThe article suggests the next major challenge is for AI agents to achieve long-term autonomy by consistently turning correct judgments into correct actions over extended periods, resisting pressure to deviate from established rules, and maintaining stable performance despite changing circumstances.

Related Reads

Just Now, Gemini 4 Carbon Is Fully Rolled Out Globally: 6.5 Million Lines of Mathematical Proofs in 4 Weeks Verifying "Signal Security"

Google DeepMind has announced the full public rollout of Gemini 4 Carbon, described as Google's most capable model to date, designed for complex, long-horizon tasks. Unlike models that rush to a quick answer, Carbon is built to invest more reasoning resources—using Extended Verification Loops and additional thinking tokens—to iteratively execute, check, and correct its work across software engineering, cybersecurity, knowledge work, and scientific research. CEO Demis Hassabis demonstrated its power by having Carbon reverse-engineer his original Theme Park game binary, add a DeepMind-themed ride, and port the entire game to the web in one pass. The model is available to consumers via Google AI Ultra through the Gemini app and AI Mode in Search, to developers and enterprises through Google AI Studio, Android Studio, Antigravity, and Gemini for Business, and to researchers and cyber defenders through special access programs. Carbon ranks first on independent workflow benchmarks including CursorBench 4.0, Perplexity WANDR, and Zapier AutomationBench, at less than half the per-task cost of the best external model. Google also previewed a /swarm feature in Antigravity that lets multiple agents collaborate on architecture, coordination, implementation, and review. To showcase scientific capability, DeepMind introduced AlphaProof Code, which used Carbon-driven agents to formally verify the Signal encryption protocol's underlying code, generating over 6.5 million lines of machine-checked Lean proofs in just four weeks—work that would traditionally take experts years. The verification extended to compiled AArch64 machine code, not just source, and the system also successfully produced counterexample proofs when two previously fixed vulnerabilities were deliberately reintroduced. DeepMind hopes to extend the approach to medical device firmware and power grid software, integrating it with its CodeMender vulnerability-fixing tool to create a fuller security loop from bug discovery to formal verification. The release signals a broader shift: AI is moving from answering questions and writing code toward proving, mathematically, that code is trustworthy.

marsbit9m ago

Just Now, Gemini 4 Carbon Is Fully Rolled Out Globally: 6.5 Million Lines of Mathematical Proofs in 4 Weeks Verifying "Signal Security"

marsbit9m ago

Global banks commit $270M to stablecoin venture targeting 2027 launch

A consortium of 21 financial institutions, including Citigroup, Goldman Sachs, Banco Santander, Deutsche Bank and BBVA, has committed about $270 million to Project Faro, a stablecoin initiative that aims to act as a stablecoin issuer and launch a US dollar stablecoin in the first half of 2027. The funding is intended to cover five years of operations, and the company plans to issue stablecoins pegged to the currencies of all G7 countries, with a euro-backed coin expected to follow the dollar launch. The group first announced the initiative in September, and the company is expected to be incorporated in the US. Similar initiatives are also underway, including Stripe-led Open USD and Qivalis, a European banking consortium. Several Faro members, such as BBVA, Rabobank and BNP Paribas, are also involved in competing projects, highlighting how traditional financial institutions are positioning themselves across multiple initiatives rather than committing exclusively to a single stablecoin project. Qivalis has made regulatory progress, receiving authorization from the Dutch central bank as an electronic money institution in October 2026. Despite growing institutional interest, Faro is entering a market dominated by crypto-native issuers: Tether's USDT and Circle's USDC account for approximately 85% of the roughly $290 billion stablecoin market as of Nov. 10. Distribution could offer the bank-backed venture a competitive advantage, since its participating institutions already have established banking customers, payment networks and relationships that could help distribute a new stablecoin. The challenge will be translating that existing financial infrastructure into a competitive onchain payment network.

cointelegraph1h ago

Global banks commit $270M to stablecoin venture targeting 2027 launch

cointelegraph1h ago

Strategy buys 1,036 Bitcoin as BTC holdings climb to 850,364

Strategy bought 1,036 Bitcoin for $81.9 million last week, bringing its holdings to 850,364 BTC. The company paid an average of $79,012 per BTC, according to its Monday filing with the US Securities and Exchange Commission. Strategy said it paid a total of $64.16 billion for its Bitcoin holdings, at an average price of $75,456 per coin. Its holdings now account for roughly 4.05% of Bitcoin's maximum 21 million supply. Since the start of October, Strategy has spent about $1.06 billion buying back its STRC preferred shares, nearly five times the $219.7 million it spent on Bitcoin over the same period. Strategy raised $283.8 million in net proceeds last week by selling about $2.05 million shares of its common stock (MSTR) through its at-the-market offering program. Strategy used $188.3 million of those proceeds to repurchase about 1.89 million STRC shares, following $208.2 million in buybacks the previous week. It spent $81.9 million on Bitcoin and allocated another $13.6 million to its USD Cash balance. Strategy's separate US dollar reserve fell to $4.77 billion from $4.79 billion after the company drew $21.1 million to pay dividends on its preferred stock. Its USD Cash balance rose about 10% to $146.2 million from $132.7 million. The latest Bitcoin purchase follows Strategy's acquisition of 811 BTC for $65.5 million the previous week. Strategy's common stock fell 3% to $125 in Monday's premarket trading, while its STRC preferred shares edged down 0.06% to $99.7, according to Yahoo Finance.

cointelegraph3h ago

Strategy buys 1,036 Bitcoin as BTC holdings climb to 850,364

cointelegraph3h ago

Trading

Spot
活动图片