Source: Wall Street Journal
Global AI infrastructure is facing a "great supply-demand fissure": demand is expanding at a thousand-fold speed, while the supply of chips, memory, electricity, and data centers will take at least three to five years to catch up. Leading venture capital firm a16z believes that against the backdrop of exponential demand growth and core component orders already backlogged for years, the traditional computing stack has been pushed to its physical limits. A comprehensive redesign of infrastructure for the AI era is now fully underway.
Recently, the top Silicon Valley venture capital firm a16z (Andreessen Horowitz) officially announced the launch of its "Machine Age Fund," focusing on investing in the AI infrastructure sector. a16z co-founder Ben Horowitz, general partner Martin Casado, and former VMware CEO and tech advisor Raghu Raghuram systematically explained the strategic logic behind this fund in a public video dialogue — and behind it lies a picture of a rapidly widening gap between AI computing power demand and supply capabilities.
The judgments of the three key figures are highly aligned: the shortage of AI infrastructure is no longer a problem of any single link; it extends from chips, memory, network interconnects, data centers, all the way to electricity, cooling, transformers, and even copper mines — "The scope of impact has extended to copper mines, that's how big this is," Raghu said.
At the same time, Ben Horowitz offered a simple but powerful judgment:
"Any problem you have can be solved with enough infrastructure — essentially GPUs plus money. As long as the problem isn't solved, demand won't stop."
Demand is "Infinite," Supply is "Fully Strained"
a16z believes that this wave of AI infrastructure boom is fundamentally different from the infrastructure bubbles of the previous internet era.
"Remember dark fiber? What was laid underground in large quantities back then was speculative, it was dark. Today, basically every GPU is pre-sold before it's even produced," Martin Casado said.
The signals from the data are equally stark:
Capital expenditures by hyperscale cloud providers are about $700 billion this year and are expected to collectively exceed $1 trillion next year.
The supply of key components is already fully booked until 2028. Martin revealed that they've even seen cases of "multi-day auctions for thousands of GPUs," with scalper prices reaching four times the original price.
Leading memory manufacturers stated publicly at the industry flagship conference Hot Chips: "The demand orders they have today would require three years of production capacity to fulfill" — and that's just the existing demand, not including future new demand.
GPU prices have also rebounded unusually. "GPU prices have always trended downward; after reaching this price, they've come back," Raghu said, noting this has almost never happened in history.
On the power side, the contradiction is equally sharp. By 2028, new data centers are expected to require an additional ~44 gigawatts of power, while grid additions are expected to be only around 25 gigawatts, leaving a huge gap. Ben Horowitz added: "We are short on GPUs, short on power, short on cooling, short on memory — whatever you name, we're short on it."
"Not an Engineering Problem, but a Resource Bottleneck"
Why does demand keep expanding without a ceiling? Martin Casado offered a core judgment:
"In the past, when you wanted to build something, it was an engineering problem — you piled on engineers, but that doesn't scale linearly; it's constrained by the 'mythical man-month' law. Now, this is truly a resource limitation problem. We pour large amounts of capital into systems, and the systems produce results. The current bottleneck is the ability of these systems to actually match the resources we're putting in."
From chatbots to reasoning models, to agents, to multi-agent systems, each generation of AI application consumes tokens at an exponentially increasing rate. Raghu's description is more direct:
"If a chat requires 100 tokens, then an agent might require several thousand tokens. Demand is expanding in two dimensions simultaneously: the number of tokens consumed per single task is increasing, and at the same time, the user base for AI is continuously expanding — from developers, to knowledge workers, to a broader range of general users."
Ben Horowitz gave his prediction: "The demand for tokens might grow close to 1000% annually. That's a speed that infrastructure supply simply cannot keep up with."
"The only way for AI to get better is to use more AI. Reasoning is the fundamental building block it repeatedly calls upon; that's why the number of tokens keeps multiplying," Raghu said.
The Entire Computing System Needs "Rebuilding from Zero"
a16z believes that the core challenge facing existing AI infrastructure is not just "insufficient quantity," but that "the entire architecture wasn't designed for AI in the first place."
Raghu systematically laid out this reconstruction logic:
"You need to examine category by category: What does the computation need? What does the memory need? How do they interconnect? How much power does each need? How to dissipate heat? How to put all these things together? This is exactly what the entire industry is doing now — decomposing the problem down to the most basic components and then reconstructing from first principles."
Martin Casado then used a business logic to justify the urgency of reconstruction:
"Training a frontier model now costs roughly $3 to $5 billion. The inference side needs at least 2x that to recoup costs, so about $10 billion. If you can save 20% efficiency on inference, that's $2 billion. And $2 billion can fully fund the development of an ASIC.
This means that custom-designing a chip for a single model is now completely economically viable. This has never happened in the history of the industry — we've never spent $5 billion to create a single digital product."
At the physical level, the pressure to rebuild is more concrete:
Rack power consumption is leaping from past levels of 5 to 10 kilowatts towards 100 to 150 kilowatts;
Compute density has increased by approximately 70x;
Cooling methods are forced to migrate from air to liquid cooling;
Rack voltage is rising to 800 volts, which is already in the high-voltage range — only about 2% of electricians in the US are certified for DC work;
Ultra-high-density racks impose entirely new requirements on data center floor load capacity;
Prices for materials like concrete are also rising rapidly.
Ben Horowitz stated bluntly:
"We have passed that era — most existing data centers will become completely obsolete once we enter the next generation of computing."
Top Entrepreneurs are Massively Returning to Hardware
This structural opportunity is already producing clear signals in the startup ecosystem.
Martin Casado revealed that a16z has an informal internal statistic: In the past, deals from top entrepreneurs involved hardware in only about 3% to 5% of cases. Now, that proportion has exceeded 20% or even 30%.
"The entrepreneur community is usually smarter than the VC community. They've identified this as a very active area of innovation and are already taking action."
The profile of these "hardware entrepreneurs" also differs from traditional software entrepreneurs. Raghu described that the founders of such companies must be "system-level founders" — "You can't just be a researcher or a good computer scientist. You need to be able to architect and design chips or systems, and also figure out how to manufacture this thing, how to set up the supply chain — things software entrepreneurs typically don't need to consider. Jensen Huang is the ultimate manifestation of this mindset; he thought about the entire ecosystem from the very beginning."
Meanwhile, frontier AI labs, due to their pressing needs, have already started signing agreements directly with early-stage hardware startups, "even before the hardware product is out." Martin believes this is an important signal of change, significantly reducing the early commercialization risk for hardware startups.
The funding environment is also shifting: the size of initial funding rounds for startups has reached the hundreds of millions of dollars level, which is extremely rare in traditional hardware investments.
From "Machine Intelligence" to "Machine Age"
Discussing the naming of this new fund, Martin Casado admitted, "The term 'Artificial Intelligence' was actually misnamed from the start — it should be called 'Machine Intelligence.'"
"It's a 70-year-old computer science term, carrying too much baggage from science fiction. And it's not truly simulating human thinking — it's leveraging everything humanity has already learned, not rebuilding language from scratch," he said.
The other layer of depth in the "Machine Age" naming lies in paying homage to hardware. "There's a deep irony here: the people who championed 'software eating the world' have finally discovered that the real limitations come from the physical machines beneath the software. This name acknowledges that in this wave, the importance of hardware is unprecedentedly significant."
Martin gave a judgment on the time scale: "We're just at the beginning — language and code, plus the nascent stage of computer use. But over the next 30, 40 years, science, materials, biology, creativity... will all be areas where computing power is thrown at problems. We are at the very early stage of a very long journey. Be prepared: this demand for computing power will last for decades."
Full transcript of the a16z interview (AI-assisted translation):
Opening Remarks
Ben Horowitz (a16z Co-Founder):
We are in the era of a completely new technology, the most important technology ever. And to support it, you need a brand new set of infrastructure.
Raghu Raghuram (Former VMware CEO / Tech Executive & Advisor):
Usually when we talk about infrastructure, it's just servers, storage, networking, and the like. But this time, it extends all the way down to mines — copper mines. That gives you a sense of how wide the impact of this is.
Martin Casado (a16z General Partner):
In the past, when you built something, the core was an engineering problem. But now, it feels like what's really bottlenecking is resources. Whether it's compute or something else, we are pouring huge amounts of money into these systems, and the systems are producing results. But the problem is, the system's ability to absorb resources simply can't keep up with the speed we're throwing it in.
Raghu Raghuram:
Top memory manufacturers say that the demand orders they currently have would require three years of production capacity to fulfill.
Host:
Ben, Martin, Raghu, welcome. Thank you. Okay, thanks. I'd like to start by quoting Marc (Andreessen) to introduce this new fund:
"This is the biggest technological revolution I have ever seen in my life. It's clearly bigger than the internet. It's comparable to the microprocessor, the steam engine, electricity, maybe even the wheel."
Folks, this is the Machine Age Fund. Please introduce it. Ben, you go first.
Ben:
Simply put, we now have a completely new technology, and it's the most important technology ever. Whenever a disruptive new technology emerges, everything we rely on — the infrastructure — needs to be completely updated along with it. And this time, the impact is deeper than ever before.
So we need not just new chips and new system software, but also completely new ways to supply power, replacing copper wire... basically, almost everything has to change. It's a very exciting time. We need a brand new investment approach for the various hardware needs of this new era.
Raghu:
I completely agree. Usually when we talk about computing infrastructure, we mean servers, storage, and networking. But this time, tracing it down, it goes all the way to mines, copper mines — that shows you how wide the impact is. That's point one.
Point two, over the past three years, we've seen the capabilities of models steadily and continuously improve, the models themselves are no longer the bottleneck. In fact, with the help of AI, these models are becoming stronger at an accelerating pace. The bottleneck now is what I call 'below the model' — the underlying infrastructure. So, that's exactly the direction we need to focus on attacking.
Martin:
I'll add this quickly: we've always followed the founders.
Over the past few years, we've observed a clear trend — more and more top teams are starting to tackle those complex, harder problems. I don't have the exact numbers, but I made an estimate myself last weekend: previously, projects from top founders involving hardware direction were probably only around 5%, but now that's over 20% to 30%.
The founder community's instincts are often sharper than VCs'. They've already identified this field as a highly active zone of innovation and are actively moving.
Ben:
I think 5% might be an overestimate.
Martin:
Right, it was indeed low, probably around 3%.
Host:
Could you explain what macro conditions are driving this shift? What is making so many founders start pouring into this direction? What opportunity do they see?
Martin:
The most obvious point is: the demand for AI is basically infinite. Because of that, the entire supply chain is under enormous pressure — including the raw materials used to manufacture things like memory.
Another special thing about AI is: because demand is infinite and growth is infinite, you usually don't need to worry about the question "can it be sold." What you really need to worry about is the company's profit margin, which is the efficiency problem.
And many efficiency problems are ultimately physical limitations of hardware. The existing systems were simply not designed for AI, nor for this type of workload. So the business model of the AI wave is putting extreme pressure on existing systems.
I think the world has realized: we must change at the core component level to improve efficiency, to support business growth, and to truly realize commercial value.
Host:
Right, so how do we know demand is truly exceeding supply now, and not just another hype cycle?
Raghu:
There are many signs that indicate this.
First, the people most capable of judging demand are placing huge purchase orders. Look at the hyperscalers, their capital expenditures have been exploding. It's said that next year, the combined capital expenditures of these players will reach $1 trillion. This year it's about $700+ billion. Think about the position of these hyperscalers in the industry — they see demand from all sides: frontier AI labs need compute, AI-native companies need compute, enterprise customers need it, the US market needs it, overseas markets need it. So if anyone has the best judgment, it's them. And their capex is rising unprecedentedly, that's a clear signal.
Second, from the companies we interact with daily, those at the application layer are all sprinting at high speed, with ridiculously fast growth rates. The same goes for frontier AI labs, it's well-documented. So I think, from the demand side, the signals have never been clearer — this is definitely not hype. Most importantly, all of this is just...
Ben:
And chip prices are still rising, we've never seen prices rise like this...
Martin:
GPU prices are falling. Prices have always been falling.
Raghu:
Yes yes yes, always falling. Look at that price curve, it went like this (downwards), then came back here. Right, and we only need 5%, 10% of the market demand to stop, and the whole market stops.
Martin:
From the supply side, existing supply is basically all booked until 2028. How bad is it — we've even seen auctions for thousands of GPUs lasting several days to close. And on the other side is demand, as Raghu said, we see the fastest-growing batch of companies in the entire industry's history.
Raghu:
Another point, the value of the work unit AI can complete is continuously increasing. But beneath the surface, the number of tokens consumed is growing by orders of magnitude. For example, a regular conversation might consume 100 tokens, but an AI Agent might consume several thousand tokens. So demand is expanding in two dimensions simultaneously: one, completing the same thing consumes more and more tokens; two, the beneficiary population in the future will be broader and broader — not just developers, but all knowledge workers, and even more. That's the trend we see.
Host:
You said key supply chain components are already sold out until 2027, maybe even 2028. What does it mean for the entire industry to be completely sold out so far in advance?
Martin:
I don't know if this has ever happened before. Do you remember? I mean, think about the internet era, we laid down infrastructure on a large scale back then, but much of what was buried underground was speculative, completely unused — remember "dark fiber"? Now, basically every GPU being produced has already been sold in advance.
Ben:
Yes, the situation was completely different back then. I mean, around 1998, 1999 there was indeed a bandwidth shortage, but the real demand for bandwidth wasn't that high because there weren't that many internet users. So it was a two-sided problem — companies were rushing in, theoretically needing more bandwidth, but on the other side, there weren't enough users consuming that bandwidth. And to truly consume a lot of bandwidth, you needed things like video, which high-bandwidth things, but video didn't work back then, for various reasons unrelated to how much bandwidth was in data centers.
So it looked somewhat similar, but it's fundamentally different. Now it's a complete shortage — people are scalping GPUs at four times the price, that's the state. Not only that, we also lack power, lack cooling. What's more troublesome is that construction itself is extremely difficult due to huge political resistance. So, in my career, I've never seen a situation like this, it's truly unprecedented.
Martin:
Let me share a small story. I recently chatted with the CFO of a large public company that has historically been very resistant to moving to the cloud, so they have a lot of their own servers. They did an inventory check and found — the value of the memory inside the servers had increased so much that just selling that memory would be enough to pay for the entire migration to the cloud. So I think we're really in a very unusual situation.
Ben:
Exactly. We're short on everything now — power, cooling, memory, GPU, whatever you name, we're short on it.
Raghu:
Yes, the industry's top flagship conference "Hot Chips" is happening at Stanford. The number one memory manufacturer said that just to meet the demand they already have today would require three years of production capacity to supply — and that's just existing demand, not even counting future demand.
Host:
So now everything is strained at the same time. Is it because everyone underestimated how strong and useful the models would be, so they simply didn't anticipate this level of demand?
Martin:
I don't think it's even that. This thing just popped up, right? It's only been four years. Even if we had perfect foresight, once it started working, I don't think...
Ben: We could have built the capacity in time.
Martin: Couldn't have built the capacity in time, completely impossible.
We're talking about chip cycles, typically three to four years. We're talking about breaking ground, building data centers, typically four to five years.
Ben:
Plus connecting to the grid, building from scratch, having a power source. So you either build your own power plant, usually you have to do both — both build your own plant and have a stable power source, which isn't easy.
Raghu:
It's like this. You have an industry that, if it can grow at 20%, 30%, that's already a very good growth rate. But now it has to interface with the AI software industry, whose growth rate is triple-digit at the minimum. You can see how big the gap is, and it's continuously widening.
Host:
So, why didn't such a fund exist before?
For example, five or seven years ago, why wasn't investing in this space as attractive as it is now? Ben, what's your take?
Ben:
I think, well, our timing now should be about right — of course, I hope so. But honestly, starting a few years earlier probably would have been completely fine too.
Martin:
I mean, you can really find a set of independent companies emerging in every technology era. Clearly, when moving from mainframes to client/server architecture, a batch of companies emerged; when moving to the internet, another batch emerged, that's why we got Cisco and Juniper. Even in the hyperscale data center era — which, by the way, was largely driven by vertical integration by a few leading cloud players themselves — you saw Arista rise.
So, there's always been an opportunity to invest along the hardware, chips line, it's just that the opportunities were relatively limited before because the change itself was also limited — maybe one chip company, one switch company. But now, everything is changing. So I agree with Ben, we could have started slightly earlier, but the magnitude of change now is just so huge, doing this is the natural thing.
Ben:
Another point is, the demand for 'intelligence' is growing almost infinitely vertically, completely unbounded. Look, companies already using AI are seeing usage skyrocket; most companies haven't deeply adopted AI yet; and the consumer side is just beginning.
So, the demand for tokens is probably growing close to 1000% annually, and supply simply can't keep up with that speed. The amount of work we need to do across the entire infrastructure layer is massive to get supply to catch up with that growth rate. So I think there are plenty of investment opportunities along this path.
Also, the architecture of existing hardware systems was designed for a completely different computing era. So it's not just "we need more compute," but we need to build new types of infrastructure from the ground up, and there are huge opportunities there.
Raghu:
Right, these technologies have all hit the physical limits they were originally designed for, like the copper wire issue Ben mentioned. You go through each technology category, and you find — okay, this technology is maxed out. So now technological breakthroughs are necessary to move to the next level.
Host:
I want to stay on the demand side a bit longer. We've gone from chatbots, to reasoning models, to agents, to multi-agent systems. With each step forward, the number of tokens consumed to complete a task multiplies dramatically.
Ben:
Nothing likes using AI more than AI itself.
Host:
Why is this trend continuing without hitting a plateau? Do you think it will just keep going like this?
Martin:
There are a few angles to answer that. First, as far as we can see now, the way we achieve scaling is essentially through massive reasoning, which is massive tokens. Think about what reinforcement learning (RL) is — massive reasoning; chain of thought — massive reasoning; long-running agents — of course, massive reasoning. So this is just one of the paths we currently use to scale.
If you want to understand from a more macro perspective, in the past, when you wanted to do something, it was an engineering problem, you threw a bunch of engineers at it, but that's not linearly scalable. Engineering has its own natural laws, that's what "The Mythical Man-Month" talks about.
But now it feels like this is truly a resource limitation problem. Whether it's tokens or something else, you pour money into the system, and the system produces results. The bottleneck now is that these systems can't fully absorb the resources we're putting in.
So I think tokens are just one stage on this scaling curve we're on, but we no longer have a "natural regulator" like we did with engineering before. So this trend should continue, and we must build enough supply to support it.
Ben:
Simply put: Any problem you have can be solved with enough infrastructure — basically GPUs and money. So as long as the problem isn't solved, demand won't disappear. That's the challenge.
Raghu:
The way AI becomes stronger is by using more AI. Reasoning is the most basic building block it repeatedly uses. That's why tokens keep multiplying like this.
Martin:
Right, and there's that "autocatalytic effect" — using AI to create more AI, like using AI to generate GPU kernel code, is itself using more AI in the process.
So we have one way to understand it: In the past, money came in, you faced an engineering problem, usually two years, often failed, there was a natural regulator, and finally you got the product. Now, money comes in, hardware directly outputs intelligence, with nothing in between blocking it. Our only limitation now is whether we can build enough supply. That's a completely different dynamic.
Raghu:
Maybe more GPUs aren't that solid a thing either...
Host:
Right, indeed.
Raghu:
So it's like a cycle.
Martin:
As long as you have money, GPUs, and data, for the foreseeable future, you can keep scaling these things.
Host:
This is really interesting. For the past decade, I've kept hearing this sentiment permeating the entire industry — too much money is flowing into startups, we're over-investing in these companies, there's too much money in venture capital.
Ben, can you say more about what this means? Because there used to be widespread skepticism: pouring more money into an industry doesn't necessarily bring greater output. But now we're saying the opposite — to some extent, how big this market can be depends on how much we collectively invest.
Ben:
Right, look, there's one thing we all know well in the startup world: If I'm two years ahead of you, and you try to catch up by hiring 1000 engineers, you'll just blow up your company. That never works. That's the mythical man-month — nine women can't make a baby in one month. It's an iron law.
But now, that logic is reversed. It's not hiring 100,000 engineers, but taking $3 billion, lighting up a hyperscale cluster, and then — suddenly, look, Grok can appear out of thin air, or whichever, just pops up, suddenly it's real.
These leads can be bridged with money; almost any problem can be solved with money, and it actually works. This is completely different from anything we've experienced before. By the way, all of us are still psychologically adjusting to this.
Host:
ChatGPT has reached one billion weekly active users. Roughly thirty million developers are using it, accounting for a significant portion of compute demand. Considering what people are actually using it for, how should we think about current and future compute demand?
Raghu:
It's a progression, right? ChatGPT started as an everyday chat tool, then became a programming assistant for professionals. Then, leveraging programming ability, many powerful tools were developed for knowledge workers. So knowledge workers are the next major battleground. Now there are over one billion knowledge workers globally waiting in line to use it. There's a long way to go on that demand. And think about the things they do — various workflows, automation, etc. — and beyond that, there's back-office business, all run by Agents. So these things are unlocked step by step, and I'd say, each unlock increases demand by an order of magnitude. We're just at the beginning.
Ben:
Right, and now with Grok Bot, the changes happening in programming are spreading to all computer usage scenarios via Grok Bot. So we're in another wave of demand, and there will definitely be more. For now, demand seems almost limitless. We haven't even talked about embodied AI or robotics, which will be another huge source of demand.
Raghu:
Martin is the expert on this, but my understanding is that Grok Bot uses "computer use" capability, like a real person sitting at a computer continuously typing.
Martin:
I actually used it last weekend — had it help me update credit card info across a bunch of services I'd been lazy about and cancel a bunch of subscriptions. I mean, that has nothing to do with programming, that's genuine computer use.
Raghu:
You've suddenly created a batch of knowledge workers, they're just all sitting inside the computer working.
Martin:
I do think Marc Andreessen is right, the closest analogy is the steam engine or electricity, and here's the reasoning: we introduced this new thing that can be converted into labor, there are some very obvious applications now, but there might be thirty or forty years ahead where we'll throw compute at problems — any problem with a clear reward signal. And we're just starting, we've only handled language and code, just these two things, computer use is just beginning.
But what else are we facing? Science, materials, biology... and of course, creative work is also a huge application area. So, we're at a very early stage of a very long journey, and we've removed one key bottleneck — traditional software engineering. Of course, bottlenecks will shift, complexity will increase elsewhere.
But I think we're in a very early, very long era of throwing compute at problems. So everyone should be prepared, this demand for compute will last for decades.
Host:
Speaking of which, Marty, talk about Grok Bot. When we chatted offstage, you said it impressed you — of course we're involved from various angles — but what do you find most interesting about it?
Martin:
I think our industry has gone through several iterations of understanding "how AI enters our lives," right?
Early on, people thought: okay, you add some AI to a product, it's just okay, like a search box. Then you started chatting with it, and it replied, because that was the conventional thing at the time.
Then OpenAI released some things, that was earlier this year. Seeing that, I thought: okay, maybe just "better search than Google" isn't the full form of AI. Maybe make it an independent entity, but it's your extension — it can share your keys, know your passwords, directly do things you would do. So it's more like an extension of you, but also more like a person.
Then I think where Grok Bot really got it right is: no, let's think differently — what if it's actually an employee? Now you have this entity that doesn't need special access to your keys or permissions, it has its own computer, its own browser. And because these are the smartest models in the world, it can do anything an employee can do.
That's interesting because now if I want to get something done, my first thought is: Can Grok handle this for me? And often the answer is yes, even for things you wouldn't have thought of. The obvious ones aside, like managing my calendar, scheduling meetings, but there are less obvious ones. For example, I'll have it read through my emails, categorize them, I didn't tell it exactly how, but it knows to check with me before actually executing the categorization. So these things are smart enough that you can give them relatively high-level tasks, and they'll figure it out and do a decent job.
Host:
Ben, I know you've thought a lot about this, including how it operates in organizations. You're, of course, also very focused on culture. What are your thoughts on this?
Ben:
Well, I mean, if you look at our own situation, it's like having a new type of employee. And there will be many, many of these employees. We have to adapt, just like it took us many years to figure out how to work with regular human employees. Now we have this other type of employee. There's a learning curve to working with them.
They might burn massive amounts of tokens, spend large sums of money, and accomplish nothing useful. They forget things, fabricate content, perform well, perform poorly, and can pose security risks. All these aspects exist. But at the same time, they can also be extremely efficient.
So I think how to integrate them, make them work well with real human colleagues, these are things we're figuring out.
I don't want to stand here and say I have the answer — that we have a perfect loop, the entire company is fully automated, I'll slowly fire all humans because I can — that's not it at all. We're more thinking: okay, how do we make all human employees super powerful, while not blowing up the company because robots can cause big trouble if they go rogue.
Raghu:
Yes, it's really interesting. We've tried several different approaches for how to best introduce agents into the system. In the end, Martin's insight proved most effective: just treat them like people, and then push things forward. That's what we're doing now, and it's proven to be the most robust way to drive this within an organization.
Host:
I want to go back to the supply side and dig deeper into where the bottlenecks are. We talked earlier about data centers, chip architecture, system software, the infrastructure itself, these were not designed for AI. If you were to truly redesign for AI, what would it look like? What should the mental model be for thinking about this?
Raghu:
Alright. If you start from the judgment that "the original model of this infrastructure must change" — then you can look category by category, find where the value discontinuity is, and then break through the bottleneck at each link one by one.
Ultimately you want to reach a state: look at what the inference engine is doing — it consumes massive memory, continuously generates new tokens while computing. Then you can think: How do I optimize this entire process? What should the memory form be? What should the computation form be? How do they communicate? How much power does each need? Given these power requirements, how do you schedule them? Then how do you put all these things together?
This is exactly what many founders across the industry are doing now. They decompose the problem down to the most basic components, asking: What kind of computation is actually happening here? The answer is matrix multiplication. So how do I optimize my compute unit for this scenario? Then, the token generation process requires more and more memory, what's the optimal way to design this memory hierarchy? How is power consumed? Then you need to connect these things — how to connect within the same chip, across chips, across data centers? How much power is needed for each level of data transfer?
So you must decompose it end-to-end, then rebuild using these basic building blocks. That's the direction we see, and where we see opportunities.
Martin:
Let me give you an interesting mental model to understand how this landscape is changing.
Today, the cost to train a frontier model is roughly three to five billion dollars. After training, inference at least needs to recoup that cost, right? For this to make business sense, you probably need to recoup about double, meaning inference needs to generate about ten billion dollars in revenue.
Now if you can save 20% efficiency in that process, that's two billion dollars. And two billion dollars is enough to build an ASIC (Application-Specific Integrated Circuit).
So we've actually reached an interesting point: it's economically viable to custom-build an ASIC for a single model, simply because the capital invested in that model is so huge.
And unlike traditional software, which has a lot of state and is very dynamic; these models are fixed, the model weights are fixed.
We don't know if the world will actually go to "one ASIC per model," but it gives you a good mental model for how architecture might evolve — becoming more and more specialized, tailored to these massive capital investments.
I think in the history of this industry, we've never directly invested five billion dollars at that scale into a single digital product. So, this will place the highest demands on hardware we've ever seen.
Host:
Speaking of which, rack power requirements are leaping from about 5-10 kilowatts to 100-150 kilowatts. Compute density has increased roughly 70x. Cooling methods have also shifted from air to liquid, which is now a hard requirement. So, what investment opportunities lie behind this?
Ben:
First, when your rack power reaches that level, alternating current (AC) doesn't work. That's actually crazy. So you have to use direct current (DC) — by the way, DC itself requires separate cooling, and honestly, it's very dangerous. It's a bit ironic because Edison originally promoted DC by claiming how dangerous AC was, even electrocuting animals in demonstrations.
Right, that horse.
So, he wasn't wrong, just about his own stuff — DC has enormous power, that's true.
Start with electricity. I think this area will see huge changes, especially combined with cooling. We're moving from air to liquid cooling, and for any cutting-edge data center, liquid cooling is already standard, that's basically set. But the problem goes beyond that.
You know, given the current political climate, just liquid cooling isn't enough, it needs to be environmentally friendly liquid cooling. DC isn't enough either; the electricity used needs to be the kind that can supply society, not just draw from it.
Some data centers behave badly now — actually a small portion, maybe 10% — wasting huge amounts of water. Sure, online comparisons with pistachios, almonds say they use more water, but data centers can be more efficient in this regard. And some data centers are like "electricity parasites," only consuming power, not feeding back into the grid. I think that will end, and must end, because we've passed the tipping point, there's no going back.
This requires quite high-level engineering capabilities, and many haven't invested in that yet. So, opportunities are coming.
Also, if rack density is that high, there are many other issues, like the floor design must be able to bear that weight, that's not just talk, it's a real engineering problem.
Also, you need large quantities of various resources. And these things are very loud, so data center walls need to be thicker, otherwise they'll disturb people, which is unacceptable. I don't think any state will allow that.
So, the mindset many have now on architecture and design is completely outdated. Once we enter the next generation, only a very small portion of existing data centers will be usable.
Raghu:
Indeed, everyone talks about memory prices, but one of the fastest-rising cost areas is actually the reinforced concrete used in data centers.
Another thing, when these data centers push 800 volts to the racks, first, danger increases significantly; second, we simply don't have enough qualified electrical contractors certified to handle 800-volt high voltage inside data centers, because that's already high voltage.
Ben:
Only 2% of electrical engineers or electricians in the US are certified for DC work. That number says it all. Meta now has a dedicated training program, training people for free to do this job, which is great, like a new version of the "Job Corps."
It's funny — they say AI will take all jobs, but AI is actually going to create a whole bunch of new electricians.
Host:
Yeah, it seems we're doing something big here.
Raghu:
Right, those giants with large cloud data centers are now frantically experimenting with robots to do tasks like assembling servers, installing equipment in data centers. As AI continues to evolve, you'll see this trend become more and more pronounced.
Martin:
By the way, the fund we're raising focuses on computer science infrastructure — meaning anything a model relies on to run, this falls under computer science. Think chips, networking, storage, all the way down, maybe extending to power itself.
Host:
What about in robotics, especially Arm architecture? Or how should we understand the direction of domestic US manufacturing?
Martin:
Of course, speaking of that — we believe any platform AI will run on is within our purview. A major breakthrough of AI is enabling computers to interact with the physical world — they can see, hear, speak. That means new platforms will emerge.
People like to say "edge devices," but that term is somewhat meaningless; it could be mobile devices, CDNs, laptops, or a physically embodied device that can move around.
So, as infrastructure investors, we don't do heavily regulated or overly vertical industries, but any computer science platform that can further extend AI outward, we are very interested in.
Host:
Back to the data center topic — by 2028, new data centers will need roughly an additional 44 gigawatts of power, while the grid is expected to add only about 25 gigawatts.
Ben:
Wait, we keep saying "gigawatts." It's like, oh, I need 100 gigawatts. Martin, what exactly is a gigawatt?
Martin:
I mean, how big is it? It takes up the space of several football fields. It's truly massive. Equivalent to the power for 50,000 people.
Ben:
So what's the power equivalent?
Martin:
What are you asking? It's equivalent to 50,000 households. Fifty thousand, 50,000 households, 50,000 homes. I grew up in Flagstaff, Arizona, a town of 40-50,000 people, maybe 60,000 with college students, our entire town's power consumption was less than one gigawatt. So you can basically use one gigawatt...
Ben:
...power your whole city, run air conditioning, that's the amount. Right.
Martin:
I mean, that's really it.
Ben:
He says it so casually.
Martin:
No no no, but to be fair, everyone throws around gigawatts, but truly operational gigawatt-scale data centers are few and far between. We have a long way to go.
Host:
So why can't power companies and big tech build faster?
Ben:
Oh, there are so many problems here. First, building data centers still relies on manual labor, so there are the usual construction issues. But what's more troublesome is you need to get permits, and also connect to power. So you need to do several things simultaneously: one, fight for power interconnect rights, which is a huge regulatory and bidding battle. Resources like natural gas grid connections are very limited. Also, you have to build your own power plant. And guess what? Now transformers, turbines, everything you need is in shortage. So you have to secure all that. This isn't a software problem, not something you solve by hiring more engineers and working weekends — that method never really worked well, but here it's completely inapplicable. There are real bottlenecks, lead times are hard to compress. Look, the world's smartest people are trying to compress time, but it's really difficult. And demand shows no signs of slowing. We're already falling behind. Demand is now growing 10x annually, supply simply can't keep up with that speed.
Martin:
By the way, how bad has it gotten — many new companies wanting GPU clusters are now going to Mexico, Australia, or other countries, simply because it's too hard to do in the US.
Ben:
Right, we're essentially handing over a lot of jobs and long-term economic benefits to other countries because we're restricting data centers in various ways here. I think the right approach should be to set a standard: data centers must give back to the community — provide more stable power supply, no noise, no water issues, and create jobs. That should be the standard. Then everyone follows it. That said, some data centers already do this, it's not some distant fantasy. Energy rates are declining every year because they generate their own power, sell electricity to the grid during the day, borrow from the grid at night — because grid usage is low at night, while power plants always generate at peak capacity. Data centers have stable power usage 24/7, and cities have high usage during the day, low at night, creating a complementary symbiotic relationship between the two.
Host:
From a big picture, we've been discussing this name, why do we feel "Machine Age" is particularly fitting to describe what we're experiencing?
Martin:
Alright, let me talk about this. First, I think Ben is completely right — "Artificial Intelligence" is the wrong term. We shouldn't have called it "Artificial" Intelligence; it should be Machine Intelligence.
Host:
Expand on that, why?
Martin: Because it's not necessarily how humans think, right? It's an archive of human thought, a collection of existing human knowledge. But so far, we don't know how to put an AI with zero knowledge into the world and have it rebuild language from scratch — we haven't done that. What we've done is: built a system that can learn from existing human knowledge and apply it effectively. And "AI" itself is a 70-year-old general computer science term covering many different things. Of course, it also carries a lot of baggage, whether from science fiction or writings from people like Nick Bostrom, etc. So point one is acknowledging a reality: this is essentially machine intelligence. Then you also need to particularly emphasize the "machine" part. There's a deep irony here — the people who championed "software eats the world" have truly reached a point where you pour in huge amounts of money, and what bottlenecks you in the end is the underlying physical machine. So I think this name is partly a tribute: in this wave, the importance of hardware is extremely significant, we want to say that.
Raghu:
Right, I think the next wave of breakthroughs will come from the quality of the underlying machine, that's basically where this name comes from.
Host:
And the name sounds cool too.
Ben:
"Machine," sounds powerful. Has a futuristic feel.
Host:
Yes... Given how large the investments in AI infrastructure are currently, and how capital-intensive these businesses are, have we passed the point where new companies can meaningfully enter the market? I mean, why wouldn't existing giants like NVIDIA, CoreWeave just take the lion's share of these markets?
Raghu:
They're all doing very well, no doubt. But going back to our earlier discussion — you don't need fundamentally new innovation to sustain growth, or the pace of improvement, whether it's tokens per dollar per second, tokens per watt, tokens per rack, or power consumption. Take any metric, if you want to achieve a 10x improvement in those metrics, you must have new innovation. And new innovation traditionally comes from founders who think from first principles, think differently, right? That's precisely what's needed for the next leap in innovation.
Martin:
That's the law of the market, right? I mean, assuming existing chip giants are already valued at trillions, which is absolutely true. Even just 5% of that is a huge private company. You could say, NVIDIA could do these things too, they certainly can, but if their focus is on things that bring 90% growth, why would they bother with the 5%? You asked the same question during the cloud computing era, right? Like, won't Amazon do this? You asked the same question during the Microsoft era, why wouldn't Microsoft do this? It's a very natural market law: once a certain scale is reached, huge innovation opportunities emerge at the edges.
Ben:
Right, our partner Alex Rampel has an interesting saying. He had a startup called Trial Pay and wanted to sell his service to Meta (then Facebook). Dan Rose, who was in charge of corporate development at the time, said: "Alex, this is great, sounds like you can pick up a lot of silver bricks, but right now I have gold bricks all over the ground, more than I can possibly pick up. The last thing I want to do is look at a silver brick." I think that's the situation NVIDIA is in now.
Host:
One hundred percent agree. We also talked earlier, if you happen to be in the sweet spot of OpenAI, Anthropic's capabilities, i.e., their core focus areas, that might be an uncomfortable position. But outside those three to five core areas, maybe...
Martin:
When a market expands, it fragments, this always happens. Think about Ford in the early days, 1913 had the River Rouge plant, it was like, coal, water, rubber trees go in, cars come out, like an assembly line.
Ben:
Right, he even bought entire rubber tree plantations in the Amazon jungle, right. There's a book called "Fordlandia" about this — he wanted full vertical integration, so he built a city called Fordlandia in the Amazon jungle, all American-style, with bandstands, ice cream, everything. And it actually ran for a while until he demanded workers show up on time, then people said, screw you.
Martin:
So now look at the auto industry, there are multiple tiers of suppliers, a bunch of companies, this always happens. Markets expand, then fragment; when growth slows, they tend to consolidate. Consolidation happens either through M&A or new challengers rising. That's the eternal cycle of private markets.
Ben:
Yes, because use cases keep multiplying, even if you're the biggest company, you can cover the largest use cases, but there are just too many use cases, and as Martin said, many are very valuable, big companies simply can't do every one well.
Raghu:
Even the architecture used to be a simple unified architecture, not anymore, it's become extremely complex. So optimizing in different ways is an inevitable trend.
Martin:
By the way, one interesting thing many don't fully understand — profit margins somewhat naturally "fall out" from standard technology approaches like software. It's not really a technical problem; once your business works, margins tend to be quite good, whether it's packaged software or SaaS models. But AI doesn't necessarily follow this pattern; we might be entering a new era where optimization at the hardware level becomes crucial for business profits in a way we've never seen before. So there are huge opportunities here.
Host:
Let's dive deeper into what kind of companies we'll invest in. Maybe start by introducing the sub-sectors, or talk about a few cases we've already invested in. I know some projects aren't announced yet, but Raghu, would you start?
Raghu:
Sure, as we've been discussing, the sub-sectors actually cover many categories. The most obvious is compute chips, but now just making a chip isn't enough; you need to deliver a complete system. What does a system need? It might involve memory innovation, networking innovation, power chips, etc. Each category could give rise to public company-level enterprises; these are all areas we're watching. After integrating all these, you also need a lot of software to support — for automating these processes, managing these equipment clusters, etc. So software is also an important area. These things are interconnected; no single category can be ignored.
Host:
Can you talk about how these companies differ from typical ones? Looking at our announced investments, the initial funding rounds are very large, often hundreds of millions of dollars. Is it a different founder type? Or what other differences? Compared to our traditional software investments, how are these businesses built and invested differently?
Ben:
I think the biggest difference, you already said it — is needing to pour in huge amounts of money before having a product. That's its nature. It's similar for large models, but I think the path for large models is relatively clearer, while this area has higher risk, requires more money, and is more complex than many things we've done before. Another point: many founders in the chip space are "veterans," like those who know how to make memory, they're not young. So that's different too, but also exciting.
Raghu:
Another point: these founders all have to be system-level founders. I mean, you can't just be a researcher or an excellent computer scientist. You need to be able to architect and design chips or the entire system, and also figure out how this thing actually gets manufactured — who supplies, and a bunch of downstream issues. If you're in software, you typically don't need to consider these at all.
The best founders — of course, Jensen (Huang) is the 'Michael Jordan' of this — thought about the entire ecosystem before even starting chip design, because there are so many bottlenecks in this industry, every link must mesh together. This is a distinguishing characteristic of these founders, and quite different from other fields.
Martin:
Also, two external environmental factors are important. First, AI labs are so starved for resources now that they're willing to proactively partner with startups. So we actually get a lot of signals quite early — labs signing deals with companies even before hardware is delivered. That's completely different from five years ago when you couldn't possibly sell an unproven hardware product to a company like Google. Second, capital supply is much looser. There's a general belief now that this is the time to reshape the entire industry landscape, with a lot of capital available for follow-on rounds. Of course, you want to invest in areas where capital is willing to go. So the overall macro environment is also completely different.
Host:
Patrick Carlson said a few years ago he felt young founders are becoming rarer — not like the days of Zuckerberg building Facebook in a dorm or Gates building Microsoft. Of course, there are still young people like Michael building iconic companies. But as you said, now it seems more 'veterans' are starting up, fewer 20-year-olds. What's your take on this phenomenon?
Ben:
I think, as Raghu said, if what you're building has an extremely complex supply chain, requires manufacturing, and is technically difficult, then experience is really important. Look at Elon or Travis Kalanick, what they did when young were also software companies. Even these top people "graduated" to tackling more complex, massive fields after accumulating a lot of entrepreneurial and technical experience. These fields have too many moving parts.
When you're learning how to start a company, it's already hard enough if you're completely familiar with the product. If you're not yet familiar with the product and have to learn while starting, for a brand-new entrepreneur, that learning curve is just too steep.
So I mean: On one hand, you have young, brilliant people like Michael who does pure software AI; on the other, you have experienced people like Elon or Travis who can navigate more complex fields. Michael might be able to do it in another 10 years, but for now, it's just too hard for him.
Martin:
Another important point — this field was neglected by the entire industry and academia for nearly 20 years. It existed, but was never a growth area. Growth was always in software, networking, those directions. So now, people graduating from university or leaving big companies with experience in this are really few and far between. You don't go for an internship and "casually make a chip."
But that's all changing. We're going to cultivate an entire generation of founders coming out of these new companies who know how to do this. They'll enter the industry in more junior roles and grow.
Speaking of which, I think one of Elon's greatest legacies isn't just the companies he founded, but the entrepreneurs coming out of SpaceX — they're changing the entire industrial system, an impact possibly exceeding the companies themselves. I believe we'll see the same thing happen in computer science and hardware.
Raghu: Yes, in fact, one of the companies we invested in was founded by two twenty-somethings, but if you walk around their office, you'll also see many experienced people. So it's a good mix.
Ben: It doesn't necessarily have to be the founders themselves with experience, but the founder must be able to mobilize and leverage experienced people, truly bring them in, and work with them, and they need to be good enough, and so on. It's quite complex.
Host: Speaking of experience, we're launching a large new fund this time, but no new general partners are joining. Right, we're sort of consolidating in a way. This is because you all, and others on the team, have accumulated a lot of experience in this field, just haven't fully unleashed it before. Right, been in a dormant state.
Martin:
It's a bit funny to say, I think we almost need someone to remind us not to go off-track, because of our backgrounds, it's quite hard. I think we need this reminder because we've spent so much of our careers on systems and hardware, naturally drawn in that direction. So look, over the years we've clearly invested quite a bit in hardware too, right? We invested in SpaceX, invested in Android, these were very early checks, we also invested in Astronomer, invested in Waymo. So even early on, we did a lot of these investments. It's just deeply ingrained in our DNA. So I don't think we necessarily need to expand the team or specifically cultivate some capability this time; these are all extra pluses.
Host:
Thank you all for participating in this closing conversation for the Machine Age Fund. Thank you, thank you, thank you.
