AI: Also to the Edge

... Nvidia, Apple very focused

Much of the discussion around Foundation LLM AI innovation has impressed us all of the extraordinary progress being made at scale with large Cloud ‘Compute’, and oodles of Nvidia’s multi thousand dollar GPU chips. OpenAI’s ChatGPT in particular has transfixed the world on the possibilities. 

But the large end is not the only place where the AI action is likely to be. It’s also going to be at the ‘small end’, or specifically ‘AI at the Edge’, on local devices, in people’s hands. And Nvidia in particular has been focused here as well. Here’s how they describe it:

“Edge AI is the deployment of AI applications in devices throughout the physical world. 

It’s called “edge AI” because the AI computation is done near the user at the edge of the network, close to where the data is located, rather than centrally in a cloud computing facility or private data center.”

They go on to add:

“Thanks to the commercial maturation of neural networks, proliferation of IoT (Internet of Things) devices, advances in parallel computation and 5G, there is now robust infrastructure for generalized machine learning. This is allowing enterprises to capitalize on the colossal opportunity to bring AI into their places of business and act upon real-time insights, all while decreasing costs and increasing privacy.

We are only in the early innings of edge AI, and still the possible applications seem endless.”

It’s why in our AI Tech Value Stack below, we’ve highlighted not just Nvidia but also Apple, even before it’s new Vision Pro platform (in Box 2), which has over two billion devices user hands with miniature GPUs as part their unique portfolio of Apple Silicon chips, capable of running machine learning and AI (Boxes 5 and 6). This approach also provides new Reinforcement Learning Feedback loops that can meaningfully augment LLM AI models small and large. 

This got highlighted with this report on Nvidia’s acquisition of OpenML, a company focused on smaller AI chip designs at the local  end. 

“Nvidia Acquired AI Startup OmniML, That Shrinks Machine-Learning Models.”

As I’ve highlighted before, Nvidia has a track record of investing early in hardware and software to become the overnight success it’s become. 

Chips intrinsically have far longer lead times, and so a proactive approach is critical. As highlighted before, Nvidia’s Founder and CEO Jensen Huang, has now a multi decade track record of planning and investing ahead. For example, he highlighted the importance of AI ‘Eating Software’ years ago, and its pivotal multibillion dollar acquisition of Mellanox, is paving the way to rebuilding AI data centers and networking today. 

Running AI applications and services in local devices takes advantage of unique data pools at the Edge, even though there are bigger power and performance issues with local AI Chips. 

And there is a lot of innovation underway in this area, both by open source companies like ggml.ai, and many others, as well as large established players like Nvidia and Apple. Will have a lot more on this front in future posts. Stay tuned.