AI: Power the next AI bottleneck. RTZ #333

...'Big and Small AI' will need huge power at both the Cloud & Local 'On-Prem' Ends

The Bigger Picture, Sunday, April 20, 2024

Sometimes it does feel like ‘Star Trek’ presciently predicted most of the promises and perils of ‘magical’ technologies, including the critical need for more Power. By now most people understand that AI GPUs and related data center infrastructure is a big supply constrained bottleneck at this stage of the AI Tech Wave.

And that Power, along with AI GPUs and the Fabs to make them, as well as a growing supply of trained AI talent are also big potential bottlenecks, as both AI hardware and software grow an exponential 10x plus a year in capabilities for the next 3-5 years alone. Of course the Government has ramped up efforts to help on Chip Fabs and Infrastructure. The Bigger Picture I’d like to delve into this Sunday, is the scale and shape of the Power demands ahead for AI. Let me explain.

This AI-driven demand for Power comes of course BEFORE the expected strain on the power grids the world over, BEFORE the needs of the Electric Vehicle (EV) industry. All in the context of a world transitioning from fossil fuels to alternative clean power sources like solar, wind and nuclear.

Of late, every CEO of major US tech companies has had this issue top of mind, as they position their companies for the AI Tech Wave to come, up and down the six boxes in the Tech Stack.

From Nvidia founder/CEO Jensen Huang (Boxes 1-5), to OpenAI/Microsoft (Boxes 2-6), to Meta (Boxes 2-6), to Google (Boxes 1-6), to Amazon AWS (Boxes 2-6), to Elon Musk’s Tesla/X/xAI (Boxes 2-6). Even Apple as it begins its AI ramp up in earnest in Boxes 1 through 6 from Apple Silicon in Box 1 to the Vision Pro in Box 6.

All of them have billions of users to serve worldwide, and millions of businesses large and small.

I’ve discussed the power demands of the AI driven work loads on the AI infrastructure side to come. As the industry harvests ever deeper and creative forms of training and inference Data (Box 4 above).

It was particularly highlighted by Meta founder/CEO Mark Zuckerberg, in an interview with tech podcaster Dwarkesh Patel’. Especially in the context of the company’s upcoming Llama 3 LLM AI models and its ‘Meta AI’ smart chat services:

“Well, there are going to be different bottlenecks. Over the last few years, I think there was this issue of GPU production. Even companies that had the money to pay for the GPUs couldn't necessarily get as many as they wanted because there were all these supply constraints. Now I think that's sort of getting less. So you're seeing a bunch of companies thinking now about investing a lot of money in building out these things. I think that that will go on for some period of time. There is a capital question. At what point does it stop being worth it to put the capital in?”

“I actually think before we hit that, you're going to run into energy constraints. I don't think anyone's built a gigawatt single training cluster yet. You run into these things that just end up being slower in the world. Getting energy permitted is a very heavily regulated government function.”

Of course, the leading cloud data center company Amazon AWS has already started signing on for nuclear powered data centers that are going to cost hundreds of millions and up per data center. As Nuclear newswire highligted in ‘Amazon buys nuclear-powered data center from Talen’:

“Talen Energy announced its sale of a 960-megawatt data center campus to cloud service provider Amazon Web Services (AWS), a subsidiary of Amazon, for $650 million.”

“The data center, Cumulus Data Assets, sits on a 1,200-acre campus in Pennsylvania and is directly powered by the adjacent Susquehanna Steam Electric Station, which generates 2.5 gigawatts of power.”

This is the nation’s 6th largest nuclear power plant, generating enough power for barely 2-3 AI data centers of the future. And we’re likely to need AI data centers that will needs gigawatts of power per data center. One sees the scope of the scaling up Power challenge. And thus why it is top of mind for CEOs and government leaders.

While on nuclear power, viewed from the US/China geopolitical tech competition filter, China is currently the world leader in new nuclear plants coming online in less than a decade. They have over 20 plans with over 20 Gigawatts of planned nuclear capacity. India is number two. The US is meaningfully behind on that ramp for now on the nuclear plant front, as CNBC notes:

“The United States currently has one nuclear reactor under construction, the fourth reactor at the Vogtle power plant in Georgia, which will be able to generate just over 1 gigawatt. (For the sake of comparison, a gigawatt is about enough to power a mid-sized city.)”

The good news though is that the US has a good stock of existing nuclear plants vs others:

“The United States' existing fleet of nuclear reactors is a testament to its prior dominance.”

The United States has 93 nuclear reactors operating with capacity to generate more than 95 gigawatts of electricity, according to the IAEA.”

“The country with the next most operating nuclear reactors is France, with 56 and a capacity for generating more than 61 gigawatts, according to the IAEA. China comes in third with 55 operating reactors and capacity of over 53 gigawatts.”

But in new nuclear plants to come, China for now has the lead.

Nuclear power aside, what I wanted underline in a more granular way today, was how the iabove mentioned billions of users and their AI smart agents, will drive the ‘Inference’ Learning loops above, in massive amounts of Math driven ‘Correlative Computing’. (The loops on both sides of the Boxes in the AI Tech Wave stack chart above).

All making the AI more useful in its ‘reasoning’ and ‘creative’ capabilities, while dialing down (or up) AI ‘Hallucinations’. And the billion quadrillion plus per second massive math inference and training correlations being run off trillions of user queries.

This is what is uniquely and exponentially driving up the power demands of the ‘Compute’ in the AI data centers in the cloud, and locally on ‘on-premises’ computers, workstations, smartphones and an exploding array of AI driven devices by Meta, Apple and many others.

It’s something Intel’s CEO Pat Gelsinger discussed at length a few days ago, saying “with Gaudi 3 and Xeon 6, AI workloads will come our way”. He was referring of course to Intel’s next generation AI enhanced chips to go up against Nvidia’s white hot in demand H100 and up AI GPUs, which are the chip of choice in AI cloud data centers everywhere. Driving PC companies up and down the ecosystem to focus on ‘AI PCs for the enterprise and consumer markets. From Qualcomm, to AMD to Intel and many more.

All of them, as mentioned earlier, have billions of users to serve worldwide, and millions of businesses large and small. And that Inference driven usage is what it likely to demand on the Power grid via both what I’ve called ‘Big and Small AI’. Exponentially in the near-term, until we figure out how to optimize the massive math processing using more efficient AI GPU hardware and software.

There are a rising band of companies ramping up to compete with Nvidia and OpenAI on their AI GPU hardware and LLM AI software leads. But that will take a while, likely a few years. And they’re CEOs will all clamor for more Power at every step of the way:

In the meantime, we’re going to have figure out pathways towards more power for AI by the gigawatts, nuclear and otherwise. And of course massive amounts of capital investments around the world up front.

Up and down the AI Tech Wave stack, at both ends of the Big and Small AI barbell. Especially as ‘Small AI’ gets way Bigger. Power is the next bottleneck to be navigated and managed. Before Scotty can beam us up to AI Nirvana. The is the ‘Bigger Picture’ this Sunday. Stay tuned.

(NOTE: The discussions here are for information purposes only, and not meant as investment advice at any time. Thanks for joining us here)