With the same one megawatt of power, NVIDIA's latest GB300 NVL72 can handle 61,400 agents simultaneously, while the previous generation H200 could only handle around 2,600.
That's a full 20 times difference.
NVIDIA's announced AA-AgentPerf scores: Under two service standards of 20 and 60 tokens per second, the concurrent agent count per megawatt for the GB300 NVL72 is about 20 times that of the H200.
When NVIDIA released these figures on June 12th, the initial external reaction was another performance flex.
But what truly changed wasn't how powerful this generation of chips is, but the ruler used to measure compute power.
It is the new benchmark released by independent evaluation agency Artificial Analysis: AA-AgentPerf.
In its official blog, Artificial Analysis called it the industry's first inference benchmark specifically designed for "AI agents."
Its main metric is also different from before: not tokens per second, but "Agents per Megawatt."
More colloquially, it's how many agents the system can simultaneously "sustain" for every 1 megawatt of electricity supplied.
We've measured FLOPS for years, and tokens per second has been used just fine, so why introduce this new AA-AgentPerf benchmark?
The Old Ruler
Can't Measure Agents Anymore
To answer this, we first need to understand what the workload really is when an agent runs.
Artificial Analysis's judgment is clear: by 2026, the most mainstream AI workload is no longer the thing those old benchmarks were designed to target: old benchmarks measure fixed-length synthetic requests and conveniently turn off optimizations that are actually enabled in production environments.
NVIDIA also gave an apt analogy:
An ordinary conversation is a 100-meter dash; the model receives a question, spits out an answer, done. But an agent working is more like a relay race.
It breaks a goal into dozens or hundreds of steps: reading files, writing code, running commands, checking results, then deciding the next step, one baton pass after another, until the task is truly done.
Along this journey, dozens or even hundreds of large model calls are strung together, each passing an ever-growing context to the next, mixed with tool calls like compilation, database queries, and searches.
The complexity isn't simply additive; it's multiplicative.
NVIDIA uses a "relay race" metaphor for agentic workloads. A goal is broken into dozens or hundreds of steps, with large model calls and tool calls passed like batons, forming a long, ever-lengthening chain.
The problem lies precisely here.
Existing inference benchmarks on the market measure single calls: how long one request takes to return, how many concurrent requests a machine can handle.
They weren't designed for agents. Chained calls, tool waiting, context inflation—the way these things stress a system is completely different from single requests.
Long sessions alone hide blind spots in old benchmarks: the same long prefix appears round after round; whoever can cache it, avoiding recomputation each time, saves a huge amount of compute.
Add to that tool outputs suddenly inflating the context, while outputs themselves are often only a few hundred tokens—whether the scheduler and memory hierarchy can handle this irregular rhythm directly determines whether a system runs smoothly or crashes.
This is precisely where fixed-length synthetic tests cannot reach.
For those spending real money on cards and building data centers, what they truly care about is how many working agents the system can sustain simultaneously, and how much useful output each kilowatt-hour and each GPU translates to.
Old benchmarks cannot answer these questions.
The First Ruler Built for Agents
AA-AgentPerf's approach differs from old benchmarks; it doesn't feed fixed-length synthetic prompts but replays real programming agent traces.
Illustration of an agent trace replayed by AA-AgentPerf. Starting from one request, LLM calls and tool calls alternate, advancing until the task is truly complete.
These traces are accumulated by having agents solve problems in real code repositories, covering over 12 programming languages. A single session can run up to 200 rounds, easily pushing context beyond 100,000 tokens.
Input lengths range from 5,000 to 130,000 tokens, averaging about 27,000. What truly inflates the length isn't the prompt itself, but the accumulated tool outputs and conversation history over multiple rounds.
More crucially, how it calculates the score.
It doesn't chase extreme concurrency. Once concurrency is stacked too high, each agent becomes slow as a crawl; high concurrency numbers are just for show, not practical.
AA-AgentPerf does the opposite: first, lock in a service standard—each agent's output speed and time-to-first-token (TTFT) must meet the target. Then see how many agents the system can sustain at most while maintaining that line.
This set of constraints has a name: Service Level Objective (SLO).
This standard is also divided into tiers, from a "good enough" tier of 20 tokens per second to a "very fast" tier of 180 tokens per second, each measuring the maximum concurrency separately, corresponding to several service levels that actually exist in the market.
How the Service Level Objective (SLO) caps maximum concurrency. The green zone is the passing area. Once concurrency increases and speed falls below the threshold, the corresponding concurrency limit becomes that system's score.
It also does something other benchmarks often avoid: it allows all optimizations vendors truly enable in production.
KV cache reuse, speculative decoding, splitting prefill and decode across deployments—tricks often turned off by benchmarks—are all permitted this time.
The reason is simple: measurements taken with these optimizations turned off are meaningless.
Simultaneously, it monitors output quality, preventing any optimization from trading answer quality for concurrency. This way, every gain from hardware or software improvements can be accurately measured.
Finally, it boils down to a core metric: Agents per Megawatt. In a world where power is increasingly tight and energy consumption is cost, this metric is what buyers truly care about: from tokens per second to agents per megawatt.
20x Lead Per Megawatt
40x Lead Per GPU
In a test representing one of the strongest current frontier Mixture-of-Experts (MoE) models, the GB300 NVL72 can sustain 61,400 concurrent agents per megawatt, averaging 57.5 agents per GPU.
The control group, H200, manages about 2,600 per megawatt, with only 1.4 agents per GPU. That's about a 20x difference per megawatt and about a 40x difference per GPU.
The significance of these two numbers also differs.
Per megawatt measures how much agent capacity you get for the same unit of electricity, an efficiency calculation. Per GPU measures the service density of a single card, a hardware calculation.
Based on these numbers, you can directly calculate what scale of agent application your power budget can actually run.
The leaderboard isn't limited to NVIDIA's GB300; it includes AMD's MI355X. Single cards, whole systems, and entire racks are all put to the test.
The first batch of results revealed two clear patterns.
Pattern 1: Rack-scale systems are inherently cheaper. They can more fully split inference and spread it across more cards, surpassing single nodes in both raw compute and efficiency per megawatt.
Pattern 2: The leap from Hopper to Blackwell pushed the concurrent capacity of systems to a whole new level, not just minor tweaks.
From Single Card to Rack
A System-Level Victory
From H200 to GB300, this might look like a single-card performance leap, but in fact, it's a system-level victory.
More crucially, the GB300 NVL72 connects 72 GPUs via NVLink into a rack-scale whole.
For such massive MoE models, this is the key: the model can be fully spread out, with experts distributed across a whole fleet of GPUs for parallel execution, rather than all crammed into a single card, wasting resources.
CUDA cores made further optimizations underneath, overlapping communication and computation across experts, letting the overhead of coordinating various experts be quietly absorbed by compute power instead of piling up as latency.
TensorRT-LLM is responsible for maintaining efficiency as concurrent sessions keep increasing, such as splitting input processing and output generation into two separate tasks, each optimized individually.
Simply put, this test score is the result of hardware, interconnect, and software stack working together.
The GB300 NVL72 rack. 72 GPUs connected via NVLink into a single high-bandwidth whole—this is the hardware foundation enabling 60,000 agents to run cooperatively.
Welding 72 cards into a high-bandwidth whole, where each GPU can rapidly share parameters, KV cache, and intermediate results—this is the confidence enabling 60,000 agents to run cooperatively.
Some Boundaries That Cannot Be Overlooked
A few points need attention here; benchmark tests cannot be equated with production reality.
First, the number 60,000 does not mean one machine simultaneously runs 60,000 independent large models.
It's a simulation of concurrent sessions under the benchmark's definition. Each agent follows a pre-recorded trace; even tool calls aren't actually executed but simulated with a fixed CPU duration.
This design ensures the final result only reflects differences in compute power itself, but it cannot be directly equated with the actual service capacity deliverable in a real production environment.
Second, benchmark scores are not production service agreements.
Artificial Analysis itself states this is a snapshot of a rapidly evolving frontier; each system still has untapped potential, and scores will climb as software optimizations advance.
Third, AA-AgentPerf is currently a standard proposed by a single organization.
Whether it will eventually grow into an industry-wide accepted ruler like MLPerf is still too early to conclude.
References:
https://artificialanalysis.ai/articles/aa-agentperf
https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/
This article is from the WeChat public account "New Zhiyuan", author: ASI Revelation, editor: Yuanyu
