This might be the moment AMD has been waiting for.
Recently, Wafer AI deployed Kimi K3 on AMD MI355X. The result is that a model that originally required 16 NVIDIA B200s running across two servers can now be deployed in a single AMD server equipped with 8 MI355X cards.
More importantly, it's not just about fitting the model.
In tests with a 1024-token input and 400-token output, the MI355X achieved a total throughput of 952 tokens/s, with single-user generation speed reaching 118 tokens/s.
Calculated on a per-node basis, its throughput is about 3.8 times that of the average per-node throughput of the 16-card B200 dual-node deployment. Its cost-performance ratio also surpasses that of the B200 and B300.
The most surprising part? ROCm didn't cause major headaches this time.
The model is too big, making memory more critical than compute power
Kimi K3 boasts 2.8 trillion parameters, requiring over 1.5 TB of memory just for model weights, not counting the KV Cache needed for its million-token context.
An 8-card B200 server offers 192 GB of memory per card, totaling about 1.5 TB. This means it's difficult to even fit the model weights completely, let alone leave space for KV Cache. Therefore, the B200 solution requires two servers with 16 GPUs.
The B300, with 288 GB of memory per card, can fit the model within a single node. Coincidentally, the AMD MI355X also features 288 GB of memory. Eight MI355X cards total about 2.3 TB, allowing the model to run on a single server.
This isn't just about using one fewer machine. When a model runs across nodes, generating each token may require network synchronization. Even with a ~195 Gb/s RoCE v2 network, cross-node communication can still slow down decoding.
Leveraging its larger memory, the MI355X keeps the entire model within a single node.
From the final results, the 8 MI355X cards achieved a peak total throughput of 952 tokens/s, with a single-stream generation speed of 118 tokens/s.
In comparison, the dual-node deployment with 16 B200s achieved a total throughput of 498 tokens/s, which translates to about 249 tokens/s per node on average.
This means the MI355X's single-node throughput is about 3.8 times that of the average single-node throughput of the B200 dual-node deployment. In terms of single-user generation speed, the MI355X's 118 tokens/s is also higher than the B200's 90 tokens/s.
The B300 remains the absolute performance leader. An 8-card B300 node achieved a total throughput of 1568 tokens/s and a single-stream generation speed of 172 tokens/s, with overall throughput roughly 1.65 times that of the MI355X.
But price changes the conclusion. Wafer calculated based on assumed hourly rates of $2.5 per MI355X card, $4.25 per B200, and $6 per B300.
Under this pricing assumption, the MI355X delivers about 48 tokens/s per dollar of peak throughput; the B200 delivers about 7 tokens/s; and the B300 delivers about 33 tokens/s.
The B300 is faster, but the MI355X offers better efficiency per unit cost. For data centers needing to run open models at scale, this may be more important than simply competing for the performance crown.
Even more surprising: ROCm was largely ready to go
For a long time, the biggest issue with AMD data center GPUs often hasn't been the hardware, but the software.
A model that runs directly on CUDA might require framework modifications, operator patches, or even low-level kernel rewrites to work on ROCm.
But the situation with Kimi K3 was different.
AMD provided near-launch-day support. Wafer stated that the model basically ran directly on the MI355X, with subsequent work focusing on a few compatibility issues and performance optimizations.
One issue arose in the speculative decoding stage. Kimi K3 itself does not provide the draft model parameters required for MTP or EAGLE, so Wafer used an external block diffusion draft model.
This setup ran directly on CUDA, but in the ROCm environment, the first real request caused a scheduler error. The reason was that a function named `top_k_renorm_prob` was missing from the ROCm branch.
What this function does is not complex: it selects the top-k values from a probability distribution, zeros out the others, and then renormalizes the retained probabilities.
Wafer ultimately patched this logic using a standard PyTorch function, without needing to write a GPU kernel or redesign the speculative decoding system.
After the fix, speculative decoding improved single-stream performance by about 2.2x, improved single-stream performance under moderate concurrency by about 1.7x, and increased peak total throughput by about 18%.
More importantly, the system could achieve peak throughput under higher concurrency.
First token was too slow, solved by just adding four zeros
Of course, throughput isn't everything in inference services. Another metric directly impacting real user experience is TTFT (Time To First Token), the wait time between sending a request and seeing the first token.
Initially, the MI355X didn't perform well on this metric. For a cold-start prefill task with about 172k tokens, the MI355X needed about 51 seconds, while the B300 only needed about 23 seconds.
For models supporting million-token contexts, prefill tasks can be enormous. If users have to wait dozens of seconds or more every time they process a long context, even high decoding speeds can't compensate for the poor experience.
Wafer eventually found that the performance gap almost entirely came from one attention kernel. In an 8-way tensor parallelism configuration, Kimi K3 allocates 12 attention heads per GPU. However, AMD's faster MLA prefill kernel in AITER only supports shapes that are multiples of 4, 8, or 16.
Twelve heads didn't match, so the system fell back to a slower, generic Triton implementation.
The solution was simple: pad the 12 attention heads to 16 with zeros, call the existing high-speed kernel, and then retrieve the actual 12 heads after computation. No model architecture changes, no writing new assembly kernels—just adding four zeros.
After optimization, the stable prefill speed of the AITER MLA kernel reached about 13,000 tokens/s, compared to the original Triton fallback path of about 4000-7000 tokens/s. As a result, cold prefill time was reduced by about two to three times.
This optimization doesn't change the final decoding throughput but significantly reduces the time users wait for the first token to appear.
This also illustrates that what seems like a large software gap between AMD and NVIDIA sometimes isn't a lack of underlying capability, but rather that existing high-speed kernels haven't yet covered a specific new model shape.
CUDA's moat remains, but a breach has appeared
One test certainly doesn't prove that AMD has caught up to NVIDIA across the board.
The B200 was forced into cross-node operation due to insufficient memory; the B300 still holds the absolute performance lead; and ROCm's toolchain, framework support, and developer ecosystem still lag behind CUDA.
But open models are rapidly entering the trillion-parameter era. When models become too large to fit in a single server, memory capacity is no longer just a number on a spec sheet—it directly impacts communication costs, deployment complexity, and final throughput.
AMD's strategy of equipping single cards with more HBM is becoming a tangible system advantage.
If AMD can continue to improve ROCm's stability, expand the shape support of its high-speed kernels, and provide more timely day-one support for new models, then data centers will have to seriously consider these GPUs. They're cheaper, have more memory, offer sufficient performance, and the software no longer requires months of tinkering.
What do you think?
Reference links:
https://x.com/wafer_ai/status/2083628389903315406
https://x.com/ChiragAsarpota/status/2083864019870634151
This article is from WeChat public account "Almost Human" (ID: almosthuman2014), author: Focus on LLMs
