Who will dominate next-generation AI infrastructure? NVIDIA, AMD, or Google?

With the recent spread of Generative AI, the soaring costs of developing and operating Large Language Models (LLMs) have become a major topic of discussion. While attention has previously focused on "how to train" models, it has become clear that the "inference" phase—where large models are actually provided as services—also incurs enormous computational resources and operational costs.

Furthermore, as NVIDIA has led the GPU market exclusively, the options for hardware surrounding AI are rapidly expanding, including Google's use of TPUs, the rise of AMD, and proprietary chips provided by Amazon (such as Trn).
In this article, we will cite statements from Steeve Morin (of ZML), an expert in AI infrastructure, to explain the differences between training and inference in the AI field, the characteristics of major hardware platforms, and the outlook for the next five years in an easy-to-understand manner.

1. Basic differences between training and inference

1-1. Training: Research and development phase

Training refers to the "learning" of large language models, image generation models, and the like. It is a process of using an extremely large amount of data to optimize model parameters and improve accuracy.
Steeve Morin describes this training stage as a "phase close to research and experimentation." The speed of repeated experiments (i.e., iteration speed) is important, and the key characteristic is the need for an environment where one can "learn more and faster," such as building larger GPU clusters and using high-performance interconnects (data communication between GPUs).

1-2. Inference: Production operation phase

On the other hand, inference is the stage where the trained model is run for "actual users" (returning inferences). For a chatbot, this means returning responses based on user input, and for image generation, it means outputting images based on text input.
What is important here is stable operation and cost optimization. It requires ensuring scalability to withstand massive requests while operating 24 hours a day as a service, while also finding ways not to waste more resources than necessary.

2. Current state of the GPU market and NVIDIA's strengths and weaknesses

2-1. Why NVIDIA built an exclusive position

NVIDIA is the central entity that has driven the deep learning boom by utilizing GPU computational resources. Originally, GPUs were parallel processing units for processing many pixels simultaneously in image rendering, but this parallel structure was also suitable for neural networks that frequently use matrix operations.
Furthermore, because NVIDIA independently developed a development environment called "CUDA," and major frameworks like PyTorch and TensorFlow have become deeply dependent on CUDA, a situation was created where many researchers and engineers had no choice but to use NVIDIA GPUs. Morin says with a wry smile, "Nobody actually wants to care about CUDA, but in the end, they are forced to use NVIDIA."

2-2. The H100 bubble and concerns about the "giant GPU approach"

As NVIDIA introduces the H100 (commonly known as Hopper), the successor to the A100, and moves on to the next-generation Blackwell (also called B100/B200), the price of server-grade GPUs has skyrocketed.
Morin points out that this "super-high-performance GPU" single-track approach could cause a bubble burst in the future. In particular, he emphasizes that in inference, there is an efficiency problem where "increasing large-scale GPUs does not mean throughput doubles," and if expansion continues while the balance between price and performance is broken, "eventually the math won't add up."

3. Possibilities for AMD, TPUs, and dedicated chips

3-1. AMD aims for an advantage in "price efficiency"

AMD's GPU architecture itself is close to NVIDIA's, and hardware prices are often cheaper than NVIDIA's. Furthermore, if they can provide GPUs with large-capacity memory (VRAM), there is the advantage that multiple models can be loaded onto a single GPU during inference and operated efficiently.
However, the reality is that it has not become as widespread as NVIDIA due to supply issues (selling out to large companies) and the software stack (the "habit" of PyTorch + CUDA). According to Morin, although there are cases where "switching to AMD can yield nearly four times the performance efficiency," there are major hurdles such as software migration costs and supply shortages.

3-2. The goals of Google's TPU and Amazon's proprietary Trn chips

Cloud providers are also actively introducing their own AI-specific chips. Representative examples include Google's TPU (Tensor Processing Unit) and Amazon's Trainium (Trn) and Inferentia (Inf).
Unlike general-purpose GPUs sold to third parties, these are developed with the incentive of 'promoting the use of their own cloud,' which creates the possibility for users to 'avoid paying high NVIDIA margins.'
However, the issue of portability is significant, and the challenge remains that there are ultimately fewer use cases compared to NVIDIA GPUs.

4. The increase in inference ratio and the outlook for the next 5 years

4-1. A future where 95% is inference and 5% is training

As generative AI services advance, it is predicted that the computing resources required for inference will become overwhelmingly large. Morin points out that 'in five years, about 95% of all AI computing will be spent on inference,' and expects that the resources required for training will remain at only about 5% of the total.
Therefore, training using large-scale GPU clusters will be concentrated among a few giant players and specific research institutes, and for general companies, how efficiently they can operate (infer) pre-trained models will become important.

4-2. Is Google the 'sleeping giant'?

Morin also argues that the companies that hold all three elements—'products, data, and their own large-scale computing resources (compute)'—will stand at the pinnacle of the AI era. From that perspective, he emphasizes that 'Google is the sleeping giant that possesses all three,' holding many users and data from mobile (Android) to business (Google Workspace).
On the other hand, while OpenAI has made a great leap forward, it cannot be said that it truly holds 'compute' because its computing resources themselves depend on Microsoft Azure. When considering who will capture the final margins, the strengths of cloud owners and companies with proprietary chips stand out.

5. Approaches to further efficiency

5-1. Large models vs. small models and RAG (Retrieval Augmented Generation)

Running huge general-purpose models 'as is' is high-cost. Therefore, approaches that optimize small models for each use case (such as fine-tuning or distillation) to perform lightweight and high-speed inference are attracting attention.
On the other hand, even with huge models, using a RAG (Retrieval Augmented Generation) mechanism that dynamically incorporates external data during inference allows for the retrieval of specialized knowledge tailored to individual tasks. RAG is implemented in a way that 'searches for relevant documents using vector search or similar methods in response to a user's question, and provides those documents to the model as context (prompt).'

5-2. Shift toward 'latency-focused' and dedicated chips

The 'inference speed' of a model cannot be measured solely by the throughput of text generation tokens (number of tokens generated per second). From the user's perspective, the waiting time (latency) from sending a request to receiving a response is what matters.
If agent-based systems develop, the time the model spends 'internally repeating thoughts (calculations)' will increase, so dedicated hardware with large on-chip memory may be more advantageous than GPUs. Startups researching such 'Compute-in-Memory' architectures are also increasing, leaving room for entities that could shake NVIDIA's dominance to emerge.

Differences between training and inference
Training is close to research and development, a phase that utilizes huge clusters to run speed-focused experimental cycles. Inference is production operation, where stable operation and cost efficiency are the keys to success.

• NVIDIA's strengths and concerns
  Software 'lock-in' via CUDA and PyTorch has created a huge advantage, but the 'scaling up' route using ultra-expensive GPUs like the H100 also carries the risk of a future bubble burst.

• The rise of AMD and cloud-specific chips
  AMD can have high hardware cost efficiency, but supply volume and software support are challenges. Google TPU and Amazon's proprietary chips are also highly noteworthy, but they also face hurdles in portability.

• Inference becomes the center of the market
  Over the next five years, about 95% of AI computing resources will be spent on inference. How efficiently this is handled will determine the success or failure of a business.

• Google holds all 'three elements'
  Google possesses products (a diverse range of services), data (collected from users around the world), and compute (its own cloud and TPUs), and may ultimately build a significant advantage.

• Future Outlook: The Evolution of Specialized Chips and Model Architectures
  With the emergence of agentic AI that prioritizes latency, architectures beyond Transformers, and innovative specialized chips such as Compute-in-Memory, AI infrastructure could undergo a major reorganization over the next five years.

Morin states, "The ultimate winner in AI will not be the player that simply makes massive models even larger, but the one that achieves 'efficiency and optimization of specialized architectures'." Furthermore, the perspective that "the company that controls all three elements—product, data, and compute—and can manage costs in an era where inference is used as a matter of course" will be the resilient winner is extremely interesting.
Understanding the foundational technologies and hardware that support AI will continue to be a crucial factor determining the success or failure of business and R&D. Looking ahead to the "inference-centric era" that lies beyond the large-scale model boom, the movement to explore more efficient and flexible forms of AI operation is expected to accelerate.

Related Articles