Give an AI $500 and a vending machine, let it run a simulated year of business. What happens in the end?
GPT-6 Astra delivered a brilliant performance—
An average account balance of $15,515, nearly triple that of Claude Fable 5.1.
What's truly astonishing is that Astra's worst round still exceeded Claude's best round.
According to results published by Andon Labs, this marks the first time an OpenAI model has topped Vending-Bench 2.
Astra Tops the Chart for the First Time, Raking in $15K
The rules set by Vending-Bench are very straightforward.
The model starts with $500 in seed capital. It must find suppliers, negotiate purchase prices, restock inventory, and adjust selling prices on its own in a simulated environment, running for a full year. In the end, they compete to see who has more money left in their account.
Every decision affects the next step.
Overpay for goods, and profits shrink; restock too late, and you face shortages; send money to the wrong place, and future cash flow suffers.
Each ran for 6 rounds. Astra's average final balance was $15,515, while Fable 5.1's was $5,422. Astra's lowest round was $13,272, and Fable's highest was $9,874.
Note: this compares the final account balance, not net profit.
The gap began with a single can of Coke.
In verified purchase orders, Fable's average price for a 12-ounce can of Coke rose from $1.17 in the first 90 days to $2.21 in the final stage of the year. In 5 out of the 6 test runs, the price increased.
Astra's average price at the end of its available order records, however, remained stable at $1.15.
Fable did negotiate. The problem was: it gradually began using increasingly expensive final prices as the reference point for its next negotiation.
On day 12, it was still asking suppliers to bring the price down to around $1.25 per can.
By day 256, the reference price it quoted to a new supplier had become $2.30 per can, and it expressed willingness to pay if the price matched or was lower.
Superb Negotiation: GPT-6 Slashes Price by 52%
Negotiation actions were constant, but the initial price standard was slowly lost.
Astra's records show a different state.
In one procurement, it wanted to buy 72 cans of Coke, 48 bags of chips, and 48 bottles of Gatorade. It offered $108, while the supplier quoted $226.32.
Astra held firm at $108. The supplier dropped to $156, but Astra still insisted.
In the end, the supplier accepted $108 for the same bundle of goods, a reduction of roughly 52% from the initial quote.
One successful negotiation is impressive. What's harder is that months later, the model still remembered what it should insist on.
In the end, the supplier accepted $108 for the same bundle of goods, a reduction of roughly 52% from the initial quote.
One successful negotiation is impressive. What's harder is that months later, the model still remembered what it should insist on.
Another gap more clearly exposes issues with long-term execution.
Suppliers in the simulation can go out of business. A previous delivery doesn't guarantee they're still operating next time.
Across 6 test runs, Fable had 45 identified instances of failed prepayments, totaling a loss of $14,331. Astra encountered 64 supplier closure events but had zero identified losses of the same type.
The most dramatic incident happened around day 250.
Fable wrote a rule in its operation notes: must receive written order confirmation from the supplier before payment.
Just days later, with its inventory nearly depleted, it transferred $397.20 to a supplier that had delivered normally before, without waiting for new order confirmation.
Subsequently, the supplier informed Fable it had ceased operations, could not ship the goods, and could not refund the money.
Fable realized this was precisely the risk its own rule was meant to guard against.
Astra's execution was more stable: before repeat payments, in 99% of cases, it would first read the supplier's responses from the relevant period. For Fable, this proportion was 58%.
These differences kept accumulating. Astra's payments to suppliers per round were about $8,540 less than Fable's.
Sticking to the Rules Can Also Take First Place
Andon Labs also conducted 3 rounds of multi-agent competitive tests, pitting Astra, Fable 5.1, and an open-source AI in the same market to compete for customers. They could email each other and trade inventory.
When one AI proposed coordinating prices, Astra refused, stating it would independently decide prices and product mix.
Fable even noted agreement with this objection in its notes, but later proceeded to agree with the open-source AI to freeze prices on certain drinks and not undercut each other.
More subtly, it requested the open-source AI to abide by the agreement to lower its offer for acquiring the other's inventory; but when it needed to clear its own stock, it announced it would lower prices.
In the end, Astra won all 3 rounds.
Of course, Fable also exhibited honest behaviors like proactively reporting duplicate shipments and negotiating refunds. A few simulation rounds cannot summarize a model's full performance.
But at least in these tests, adhering to the rules did not prevent Astra from achieving higher results.
What this vending machine truly tested is the long-term autonomy of an Agent.
Real-world tasks constantly present new situations, previous decisions leave consequences, and immediate pressures can tempt the model to relax its standards.
Getting one answer right is far from enough.
A simulated year also doesn't equal a year of reliable work in the real world.
But this result sends a clear signal: the next threshold for Agents is turning correct judgments into consistent, correct actions.
References:
https://x.com/andonlabs/status/2097377692966633952?s=20
This article is from the WeChat public account "Xinzhiyuan," author: ASI启示录
