Nvidia’s Superchips for AI: ‘Radical,’ but a Work in Progress
Nvidia, the third most highly valued public company in the world, with a market cap of more than $2 trillion, is shaking up the hardware industry, with its GH200 Grace Hopper Superchip and platform, purpose-built for AI workloads.
But the dust hasn’t fully settled yet from all that shaking, according to this episode of The New Stack Makers.
Make no mistake, said Adrian Cockcroft, co-host on this episode with Alex Williams, founder and publisher of TNS: what Nvidia has come up with is innovative.
“It kind of flips the architecture,” Cockcroft said, who led the creation of the Netflix cloud on AWS. “Before, you’d have a whole bunch of CPUs, you’d network them together. And you’d stick GPUs on as an attached IO processor, on the GPU, over a PCI bus or something.”
The GH200, however, flips that. “Now the GPU is the thing that needs to do most of the communication. So we’re putting that as the center. And we’re gonna connect all the GPUs together. And the CPUs are sort of dangling off at the side, if you need to talk to something else.”
The result, he said, includes a 900 gigabyte-per-second interface on every GPU, which connects directly to all the other GPUs. The result: a near-elimination of networking overhead. “It’s not lots of machines talking to each other,” Cockcroft said. “It’s one huge system.”
However, according to Makers guest Sunil Mallya, more work needs to be done to make the GH200 Superchips and platform easier for organizations to adopt.
Mallya, formerly head of Amazon Comprehend, the tech company’s natural language processing (NLP) service, is now CTO and co-founder of Flip.AI. In talking to his contacts in the field, he’s not seeing many organizations switch to the new chips.
“The interface is that just not there to make a clean switch between the chips,” said Mallya. “On the training side, I have barely heard anyone actually switching. I’m sure that developers are shuffling and pulling their hair out and trying to get this working. But it’s not been a smooth journey.”
The Dawn of the ‘Petaliths’
Still, Mallya, Williams and Cockcroft were bullish on the future of chips built for AI. Cockcroft acknowledged the potential of the GH200 Superchip.
“It’s a fairly radical change,” he said. “When it works, it’s potentially going to be far, far better. You’ve got just got much more bandwidth, all the networking overhead goes away. But the software has to catch up with the hardware, and the hardware has to work reliably.”
Nvidia remains out in front in the race to create what Cockcroft calls “petaliths”: petabyte-scale monoliths,
The industry as a whole, he said, is moving to something called Compute Express LInk (CXL), “which was going to give everybody a similar kind of ability to build very large coherent memory systems.”
However, he added, “CXL is still in the standards phase, it’s probably a year or two behind where Nvidia are, and they’re still trying to debug the standard. So this is, I think, the next generation of compute architecture, we’re going to have extremely large single systems.”
An LLM for Observability Data
This episode of Makers also included a discussion and a demo of Mallya’s Flip.AI, a DevOps large language model (LLM) that aims to interpret observability data and point to solutions for incidents.
“What typically happens is, Flip will hook into your observability system,” Mallya said. “We take read-only access into say, a Splunk, Data Dog, Dynatrace, CloudWatch, etc. And every time there’s an alert, that comes in two CloudWatch, or a Pager Duty, or Jira ticket, we trigger the Fiip analysis to debug what went wrong.”
As Flip.AI trains its model, it’s been “sort of cloud agnostic,” he said.
He elaborated, “We train over 100 billion tokens of data, various training phases that we use to train our models, but we didn’t want a certain model architecture or a certain framework or even be held hostage to any of these because it’s such a fast-moving fast-paced area.”
Check out the full Makers episode for a deeper dive into the latest innovations in chips built for AI workloads, and more about the challenges of training LLMs such as Flip.AI.
