One Megawatt Sustains 60,000 Agents, NVIDIA GB300 Crushes Previous Generation by 20x
NVIDIA's latest GB300 NVL72 system achieves a 20x improvement in AI agent throughput per megawatt compared to its predecessor, the H200, according to a new industry benchmark called AA-AgentPerf. Where the H200 could handle roughly 2,600 concurrent agents per megawatt, the GB300 NVL72 can support approximately 61,400.
The significance lies less in raw chip performance and more in the new benchmark itself. AA-AgentPerf, created by the independent firm Artificial Analysis, is the first benchmark designed specifically for "AI agent" workloads. Traditional benchmarks measure single, fixed-length requests, but AI agents operate in long, complex chains involving dozens of model calls, tool use, and ever-growing context. These create unique system pressures that older tests cannot capture.
AA-AgentPerf replays real programming agent trajectories with lengthy sessions and varying input lengths. Its key metric is "agents per megawatt," measured under strict Service Level Objectives (SLOs) that guarantee a minimum token output speed per agent. It also allows real-world optimizations like KV cache reuse and speculative decoding, which older benchmarks often disable.
The results highlight two key trends: rack-scale systems like the 72-GPU GB300 NVL72 are inherently more efficient than single nodes, and the architectural leap from Hopper to Blackwell (H200 to GB300) represents a systemic, not just incremental, performance gain. The GB300's advantage stems from its high-bandwidth NVLink fabric connecting all GPUs, allowing large MiE models to be efficiently distributed and parallelized.
Important caveats include that the 61,400 figure represents simulated concurrent sessions, not independently running full models, and that benchmark results are a snapshot that will improve with software optimization. AA-AgentPerf is a new standard whose industry adoption remains to be seen.
marsbit07/06 01:03
