AI: Nvidia's small AI computers. RTZ #577

...for robotics and other local small AI Edge applications

The Bigger Picture, December 21, 2024

I’ve talked a lot in these pages about Small LLM AIs, even tiny ones, situated locally near users, playing a big role in this AI Tech Wave, along with LLM AIs in big, expensive AI Data Centers. Most of those discussions centered around software. But also needed will be local hardware, beyond AI chip powered PCs and Smartphones, to do a lot of the AI heavy lifting without always relying on AI cloud data centers. That is the Bigger Picture I’d like to discuss today.

It’s timely because Nvidia founder/CEO Jensen Huang unveiled the latest, local AI processing powerhouse, the Jetson Orin Nano Super Developer Kit computer, capable of 70+ teraflops of local computational power, for under $249.

As the WSJ explains in “Nvidia Introduces Device Aimed at Small Companies, Hobbyists for AI Use”:

“At $249, new version of Jetson computer is half the price of its predecessor”.

“Nvidia said Tuesday that it was introducing a $249 version of its Jetson computer for artificial-intelligence applications, half the price of its predecessor, to attract more hobbyists and small companies.”

For now, these are not consumer devices, but designed for businesses to create AI driven devices and systems for edge use.

“The Jetson computers are essentially portable brains, mostly designed to allow developers of robots, industrial automation and other hardware to run sophisticated AI computations directly without connecting to a remote data center.”

“Nvidia’s main customers are big companies and AI startups that spend hundreds of millions or billions of dollars on hardware to train and operate their AI models. With its Jetson lineup, the company is making a play for accessibility, pitching the low-cost devices to small companies, hobbyists and students who want to try creating new products that integrate AI functions.”

The company went out of its way to introduce the product with a differentiated approach, exhibited at the top of this post.

“In a promotional video, Nvidia founder Jensen Huang broke from his usual stagecraft—posing before towering server racks that use Nvidia chips—and instead presented the new palm-sized Jetson device on a tray, as if pulling it fresh from the oven.”

And even in the base configuration, the device has impressive specs.

“The $249 Orin Nano Super nearly doubles the speed and efficiency of the previous device and can process about 70% more computational tasks, Nvidia said. It contains less-advanced chips than those powering Nvidia’s top-of-the-line products and targets commercial developers working on consumer technologies such as drones and cameras.”

There of course is a beefier line, Nvidia’s Thor systems.

”The higher-end Jetson Thor is designed to support humanoid robots and sophisticated automation.”

“This is the time, finally, when generative AI capability is coming to the edge,” said Deepu Talla, Nvidia’s vice president of robotics and edge computing. “Edge” refers to individual devices in factories and labs as opposed to the large computer centers where most advanced AI work has taken place to date.”

A key application of these devices, is for AI Robotics, a big, emerging application for foundation large and small language models. They leverage Nvidia’s GR00T Foundation Models, and Isaac Robotics Platforms. These are part of Nvidia’s formidable software moat for AI applications across industries.

“Analysts said the Jetson line could diversify Nvidia’s portfolio, in particular by drawing customers who are looking to develop robots.”

Nvidia in particular is targeting Jetson Thor computers for Humanoid robots next year, a key target market for CEO Jensen Huang. As the WSJ notes in a separate piece:

“Nvidia is set to bring its new technology for powering humanoid robots to market in the first half of 2025, aiming to stake its claim in the fast-growing robotics sector, a senior company executive said.”

“First unveiled earlier this year, the Jetson Thor computers are part of Nvidia’s approach to developing humanlike robots, where advances in artificial intelligence have improved autonomy, enabling robots to interact better with humans and their surroundings.”

“Jetson Thor is the latest addition to Nvidia’s Jetson platform, a line of compact computers designed for AI applications, with the new model now focused on robotics.”

Nvidia of course is not alone in these local computing systems:

“Other tech companies such as Intel, Alphabet’s Google and Qualcomm offer similar edge systems. They could challenge Jetson’s position in edge AI by offering plans tailored to applications such as vision processing.”

“Asked about rivals, Talla said Nvidia’s product was for general purposes and could run “all the latest greatest generative AI models.”

And as I’ve noted before, China remains a formidable competitor in AI powered robotic systems.

“While the U.S. has restricted Nvidia from selling its most advanced hardware in China, the company said the new Jetson product would be available in China through local distributors.”

We’re going to need AI systems of all sizes, in hardware and software, to make really interesting applications and services happen at scale in this AI Tech Wave.

And that is the Bigger Picture to keep in mind as we wrap big and small packages this Sunday for Wednesday. Happy Holiday Weekend. Stay tuned.

(NOTE: The discussions here are for information purposes only, and not meant as investment advice at any time. Thanks for joining us here)