CUDA has been hit yet again...
This time, the perpetrator is the AI that NVIDIA itself helped nurture.
A small company founded just a year ago used an AI programming Agent to build a "CUDA-like software" suite in just 10 hours.
Wait, wait, wait.
Are you saying this software empire that NVIDIA spent nearly 20 years building has just been replicated??
And that's not all. DeepSeek is also trying to write less low-level CUDA code.
They have open-sourced a GPU operator library named TileKernels, written using TileLang, which saves a lot of manual low-level CUDA coding work.
However, similar "CUDA crises" are not new to our ears.
From time to time, CUDA gets dragged out for a beating, with claims of being replaced, disrupted, or having its moat overturned again...
Is it really so?
10-Hour "Replication" of CUDA
Mentioning NVIDIA, many people's first thought is GPU.
H100, Blackwell, Rubin—through generations of increasingly powerful chips, NVIDIA has captured the biggest dividends of this AI wave.
But NVIDIA's truly formidable advantage is not hardware, but software.
Chips can be bought, parameters can be chased, and process technology will iterate. What really makes customers stick with NVIDIA after they start using it is the entire software ecosystem built around the GPU.
And CUDA is the core of this software empire.
CUDA, short for Compute Unified Device Architecture, was built over nearly 20 years by NVIDIA executive Ian Buck and his team.
It is often called a programming language, but more accurately, CUDA is a complete software platform built around NVIDIA GPUs.
If simply divided into three layers: the bottom is the kernel layer, the Kernels, compilers, and runtime that directly make the chip work.
The middle is the library layer—highly optimized computation libraries like cuBLAS, cuDNN, as well as debugging, verification, and profiling tools.
The top layer includes development frameworks like PyTorch and TensorFlow, the massive codebase and workflows accumulated by enterprises, and the developer ecosystem that has grown around CUDA.
△
This might sound a bit abstract, but you can think of an NVIDIA GPU as a factory.
The bottom layer of CUDA is responsible for telling the machines exactly how to do each step, the middle layer provides ready-made production tools, and the top layer is a complete, well-oiled production system that has been running for years.
So, for customers, what they buy is not just an NVIDIA card, but a factory that's ready to use out of the box.
Now, a guy named Jeremy Nixon is here with an AI Agent.
Nixon is the founder of the AI software startup Infinity and was previously a researcher at Google Brain.
His team developed an AI research Agent named Ignition, specifically designed to write low-level software for different AI chips.
It can generate GPU Kernels, run tests, find errors, measure performance, and automatically rewrite code based on the results.
Human engineers provide high-level direction, and the tedious, brain-intensive work is left for the Agent to loop through repeatedly.
Thus, Infinity used Ignition to build a CUDA-like software suite for the AI chip startup d-Matrix.
It took only 10 hours.
Huh??
Can the low-level code that previously required chip software engineers to debug repeatedly now really be handed over to AI?
It can't be dismissed entirely as hype.
AI Agents are indeed well-suited for programming tasks with clear feedback.
Whether the code can compile, whether the results are correct, whether performance has improved—all can be answered by actually running the code.
Thus, the Agent can form a complete closed loop:
Write code → Compile → Run → Test correctness and performance → Modify based on feedback
However, Infinity primarily targeted the first layer of CUDA: generating and optimizing inference Kernels for different chips and building low-level software capabilities around these Kernels.
It is developing a general inference library for multiple chips, but this does not equate to replicating CUDA's complete ecosystem built over nearly 20 years within 10 hours.
But...
But!
You don't actually need to replicate CUDA to secure $15 million in funding.
The company's valuation has already reached $100 million.
Whether it threatens NVIDIA or not, capital is certainly buying into it. (doge)
The Second Front is Activated on the Inference Side!
If AI Agents are the weapons for the siege, then the inference market is the gap in the wall.
Large model computation is mainly divided into two stages:
Training is building the model, requiring tens of thousands of cards to run collaboratively for months; Inference is using the trained model to answer questions, which can run on small clusters or even a single card.
On the training side, CUDA currently remains nearly unbeatable.
Large-scale training is extremely sensitive to performance, stability, and cluster coordination. Communication between chips, memory scheduling, recovery from program errors—any loss of efficiency, amplified across thousands or even tens of thousands of chips, translates into real time and cost.
Therefore, enterprises are often extremely conservative when choosing a training platform.
Even if other chips have good paper specifications, if the software isn't mature enough or the cluster isn't stable enough, the hardware cost saved might be paid back twofold in development and trial-and-error costs.
But on the inference side, the rules of the game have changed.
Marshall Choy, Chief Business Officer of the Korean AI chip company Rebellions, believes:
On the inference side, CUDA is no longer the deciding factor. This will be a competition of open-source software.
Why are competitors all focusing on inference?
Because training cares about "peak performance," while inference cares about "cost per answer."
Training requires tens of thousands of cards to collaborate; inference can be split into small clusters or even single cards. Training dares not switch chips, but inference can.
As long as it runs and is cheap, customers are willing to try.
More crucially, inference software can run across different chips. If an inference framework can support multiple chips, users can switch between different hardware without rewriting code—
Thus, CUDA's biggest lock-in effect becomes ineffective.
This creates an opportunity for dedicated inference chips.
Inference tasks don't necessarily require all of CUDA's capabilities from training to cluster coordination. Chips redesigned for specific models or scenarios have a chance to carve out a piece of the market from NVIDIA, as long as they can run faster, cheaper, or more power-efficiently.
Take d-Matrix, for example, a chip company focused on inference.
Founded in 2019, this company focuses on generative AI inference chips.
Co-founder and CEO Sid Sheth recalled that they judged early on that the opportunity brought by AI inference would eventually surpass that of training.
The CUDA-like software that Infinity built in 10 hours using an AI Agent serves precisely d-Matrix's inference chips.
Thus, the complete story of the "10-hour incident" comes together:
A new chip wants to enter the inference market but lacks mature software; AI Agents quickly generate and optimize low-level Kernels, compressing adaptation work that could have taken months or even years into hours or days.
Sid also explicitly stated:
While NVIDIA maintains dominance in the training domain, in inference, the moat has weakened.
d-Matrix isn't the only one eyeing this gap.
Inference-focused companies like Rebellions and d-Matrix, wafer-scale chip bettor Cerebras, Amazon's Inferentia, Google's TPU, AMD's MI300X...
Various chip manufacturers and cloud computing giants have already deployed their forces in the inference market, preparing to snatch a piece of meat from NVIDIA.
For these companies, they don't need to replace NVIDIA immediately.
They only need to prove that, for certain inference tasks, they can run faster or cheaper without relying on NVIDIA's full hardware suite and CUDA ecosystem.
That is enough.
Has NVIDIA's Moat Really Been Disrupted?
The answer might be... not even close.
As mentioned earlier, the 10-hour rewrite is for the "adaptation code of one chip," while the CUDA ecosystem still has 20 years of accumulated libraries, debugging tools, community, and millions of developers.
This is not something easily disrupted.
More importantly, code can be generated, but that doesn't mean it's usable.
Bing Xu, founder of INT21, whose previous AI chip software company HippoML was acquired by NVIDIA, left NVIDIA just this April.
His assessment is: Agents can generate a large amount of code in a short time, but verification is the biggest bottleneck.
AI can quickly generate ten thousand lines of code, but "proving that these ten thousand lines run correctly and stably under all edge cases" is extremely slow.
CUDA's deepest asset is precisely its verification toolchain—so he believes that in the Agent era, the verification ecosystem will become CUDA's next moat.
Chris Lattner, Co-founder and CEO of Modular, also poured cold water on the idea.
He believes the improvements brought by coding Agents are incremental, and "the hype is severely exaggerated."
There are three reasons: First, writing code is only a small part of software engineering; production-level optimization, which directly determines chip performance and the cost of running AI, is what matters.
Second, chip software is a niche, elite field with far less publicly available code than application development, so there's inherently less training data for AI to learn from in this area.
Third, the migration cost of millions of lines of existing code remains, which is not a problem technology can solve; it's an organizational decision.
Another easily overlooked point: NVIDIA itself is also using AI.
Ankit Patel, Vice President of NVIDIA's Developer Ecosystem, stated:
We are also using AI Agents to develop CUDA faster and to verify it at a larger scale.
Using one's own spear to attack another's shield—the relative advantage of the attackers is also diminished.
Bing Xu summarized: The outcome of this battle depends on whether competitors can catch up to NVIDIA faster than NVIDIA can iterate on itself.
But obviously, the world's largest chip manufacturer is "not sleeping or standing still."
Overall, in the short term, NVIDIA's moat remains intact, but changes will quietly begin to occur on the inference side.
The real long-term suspense is: the moat is shifting from "the stockpile of code" to "the ecosystem of verification and optimization."
Whoever secures the new position first will be the next winner.
Reference link:[1]https://www.businessinsider.com/nvidia-cuda-new-threats-ai-coding-agents-2026-8
This article is from the WeChat public account "QbitAI," author: Ting Yu
