[5/30] The Multi-Billion Dollar Challenge: The AI Infrastructure AWS is Building

In recent years, as the commercial use of AI (Artificial Intelligence) has accelerated across all industries, the strategic positioning of AWS (Amazon Web Services), a provider of cloud infrastructure, has become increasingly important. In late May 2025, Matt Garman, who marked his first anniversary as CEO of AWS, spoke about the current state and future prospects of the company's AI business. Based on that interview, this article explains the AI vision AWS is painting, incorporating specific examples and quotes. By organizing professional content in an easy-to-understand manner, we aim to get to the essence of the transformation that AI is bringing to AWS for readers including corporate executives, engineers, and investors.

1. Overview of the AI Business at AWS

1-1. AI as a Multi-Billion Dollar Business

Matt Garman reflects, "We have experienced phenomenal innovation over the past year, and our customers are moving explosively toward AI utilization." He then stated clearly that the AI-related business provided by the company has grown into a "multi-billion dollar business."

This business scale mainly includes the following elements:

• Cases where customers themselves build and operate their own models on AWS

• Cases using managed AI services provided by AWS

• AI utilization within Amazon itself (e.g., product recommendations, logistics optimization, applications for Alexa, etc.)

For example, at Amazon, AI is widely used for logistics warehouse optimization, customer review summarization, and improving search accuracy. Garman says, "We also use AWS to build AI in-house and are working on operational efficiency and improving the customer experience." As a result, the structure is one where AWS itself is strengthening its aspect as an "AI company" while providing similar value to its customers.

1-2. Major AI Services and Customer Use Cases

Representative AI services provided by AWS include the following:

• Amazon Bedrock
  A service that allows integrated use of third-party and AWS first-party Large Language Models (LLMs). According to Garman, customers enjoy significant benefits in implementation costs and scalability because they "can utilize models like Anthropic on Bedrock without being conscious of low-level chip operations."

• Amazon Nova
  A generative AI model developed independently by AWS, which has strengths in text generation and code generation. Garman introduced an internal practice example, saying, "We also use Nova in-house to automate software development and improve efficiency."

• Collaboration model with Anthropic
  In an initiative Garman calls "Project Rainier," they are building one of the world's largest training clusters in collaboration with Anthropic. Using custom AI chips called Training M2, they provide a learning environment for high-performance models, including Anthropic's latest model, "Claude 4."

• Amazon Q
  Amazon Q, which is attracting attention as an automated software development support tool, uses AI to generate code snippets and detect bugs, contributing to improved developer productivity. Garman explains, "By having enterprise customers introduce Amazon Q using Bedrock and Nova, they can significantly reduce development man-hours."

As for customer cases utilizing these services, in the financial industry, there is an example where analysis time was reduced by more than 50% compared to the past through "automatic summarization of transaction reports and risk analysis by AI." In the manufacturing industry, there are various cases, such as building an "AI-assisted design automation platform" where the design cycle was shortened to one-third of the previous time.

2. Trends in AI Workloads (Training vs. Inference)

2-1. Current Status and Future Outlook of Training

During the interview, Garman said, "Currently, most AI workloads are occupied by training, but in the future, inference usage will become mainstream." Specifically, while many companies and research institutions focused on the model-building phase until early 2024, as models have become larger and more diverse, operational usage involving inference is increasing explosively.

Supporting this view, Mr. Garman stated that "inference will become the core of enterprise applications, standing alongside storage, compute, and databases as a fundamental infrastructure element," and indicated that AWS intends to focus on strengthening inference-optimized instances and optimization algorithms.

2-2. Expansion and Integration of Inference

The background to the increase in inference workloads is the growing need to incorporate AI features into consumer applications and business systems. Mr. Garman pointed out, "Companies have already entered the stage where they want to utilize AI in production environments rather than just for research and experimentation."

For example, in call centers (chatbots and voice bots using Amazon Connect), e-commerce recommendation engines, and diagnostic assistance in the medical industry, the shift toward mechanisms where inference models operate in real-time is accelerating. In response to this, AWS is "strengthening the provision of inference-optimized instance types (e.g., Inf1, Trn1, etc.) and low-latency inference serving frameworks."

3. Token Usage and Performance Metrics

3-1. Understanding Token Growth

As an indicator of AI model usage, "token count" often receives attention. Regarding this, Mr. Garman stated, "In text generation models, costs tend to become more apparent as the token count increases, but tokens do not necessarily represent the total 'workload'." For instance, if an AI performs thousands of internal inference steps to generate a single final output, the amount of tokens processed in that process is not measured, but in reality, a vast amount of computational resources may have been consumed.

For this reason, AWS emphasizes an approach that tracks performance metrics from multiple perspectives beyond tokens, such as "the amount of computation performed internally by the model," "GPU/custom chip utilization time," and "inference latency," to provide the optimal cost-performance for customers.

3-2. The Need for New Performance Metrics

Mr. Garman suggested that the following metrics will become important in the future.

• End-to-end inference latency (response time from input to output)

• Energy consumption per computation (which has a significant impact on large-scale data center operating costs)

• GPU/chip utilization rate during the training/inference phase

• ROI (Return on Investment) evaluation based on user experience

He stated that by managing these metrics comprehensively, "it becomes possible to accurately estimate the TCO (Total Cost of Ownership) after AI implementation and quantify the value generated when compared to traditional cloud usage models."

4. Project Rainier and Training M2 Chips

4-1. Overview of Project Rainier and Collaboration with Anthropic

"Project Rainier," which was discussed in the middle of the interview, is an ultra-large-scale training infrastructure construction project being advanced jointly by AWS and the AI startup Anthropic. According to Mr. Garman, they are providing one of the largest computing clusters in the company's history to train Anthropic's latest model, "Claude 4."

This collaboration will dramatically increase the training speed of large language models while strengthening AWS's competitive advantage as a cloud provider. In return, Anthropic plans to deepen its commitment to the AI platform running on AWS and continue to pursue joint research and development.

4-2. Performance and Cost Optimization of the Training M2 Chip

Training M2 is custom silicon for AI developed in-house by AWS, which is said to deliver performance in both large-scale model training and inference. Garman explains, "M2 not only improves absolute performance (such as TeraFLOPS) but also significantly enhances cost-performance compared to traditional general-purpose GPUs." The following factors are cited as reasons for this.

• Optimization of dedicated AI acceleration circuits

• Improved data movement efficiency through enhanced High-Bandwidth Memory

• Faster inter-node communication via low-latency networking

As a result, not only Anthropic but also many other AI users, including Google, Meta, and university research institutions, are choosing M2-based clusters. Garman stated, "In AI development, the challenge of high costs still exists. However, with the arrival of M2, we want to significantly lower those costs and create an environment where more developers and startups can enter AI research."

5. Relationship with NVIDIA and Competitive Strategy

5-1. Collaborative Relationship between NVIDIA and AWS

When it comes to AI infrastructure, NVIDIA GPUs have long been the industry standard, and AWS is no exception. Garman emphasized the importance of the partnership between the two companies, stating, "NVIDIA has built a fantastic platform, and we are constantly providing their latest technology on AWS." Specifically, AWS is quick to equip its cloud instances with new-generation NVIDIA GPUs, providing customers with a "state-of-the-art AI computing environment."

On the other hand, AWS is also introducing its own proprietary technologies, such as custom chips (Training M2) and low-cost inference instances, to differentiate itself from NVIDIA GPUs. Garman pointed out that "the combination of M2 and NVIDIA is the optimal solution" and that "customers want choices." While the technologies of the two companies compete, he supports building flexible environments tailored to user needs.

5-2. Cost Performance and Competitive Advantage

While NVIDIA GPUs still have a performance advantage, they are often positioned as high-end in terms of cost. In contrast, Training M2 plays a role in lowering the barrier to AI adoption for startups and small to medium-sized enterprises by "maintaining a certain level of performance while keeping the price per unit of computation low." Garman stated, "M2 functions as a cost-efficient alternative to NVIDIA products," and introduced cases where multiple customers have reduced costs by up to 30-40% using M2-based AI clusters.

Furthermore, AWS has indicated a policy to enhance its competitive advantage in the AI infrastructure market by "continuously developing differentiated chips like M2 to prepare for the rising costs of NVIDIA GPUs, thereby providing an environment where customers can optimize their AI workloads over the long term."

6. Global Data Center Expansion Strategy

6-1. Progress in the Latin American Market

AWS is actively investing not only in North America and Europe but also in the Latin American market. Garman said, "We launched the Mexico region earlier this year and have already confirmed high demand," adding, "Plans for a Chile region are also underway, and together with the Brazil region, we want to build a system that meets cloud demand across all of Latin America." This is creating an environment where government agencies, financial institutions, and technology startups in the region can use low-latency and high-availability cloud services.

6-2. Launch of the "European Sovereign Cloud" in Europe

In Europe, there is a growing demand for a "Sovereign Cloud," particularly from governments and financial institutions, from the perspective of data sovereignty and privacy protection. In response, AWS announced that it plans to launch the "European Sovereign Cloud" by the end of 2024. Garman explains as follows.

Specifically, they are distributing data centers across regions in various European countries to strictly manage data residency and access rights. Furthermore, by incorporating local certification authorities and audit systems, they guarantee services that comply with GDPR and individual national data protection laws. This will allow enterprises and the public sector within the EU to utilize AWS cloud services with peace of mind.

What emerges from Matt Garman's interview is a strong message that "AWS views AI not merely as an added value, but as a foundation that fundamentally transforms both its own business and its customers' businesses." Organizing the content of the interview clarifies the following points.

1. The AI business is already on a multi-billion dollar scale

   • By AWS utilizing AI itself and providing a wide range of AI services to customers, the entire AI business is scaling up.

2. Shift from training to inference phase

   • Going forward, inference workloads will become primary, and the concept of "Inference as a Building Block" for embedding AI into applications will become widespread.

3. Cost optimization through the use of diverse performance metrics

   • In addition to token counts, AI infrastructure will be managed using multifaceted metrics such as latency, energy efficiency, and chip utilization.

4. Building an ultra-large-scale training foundation through collaboration with Anthropic

   • In "Project Rainier," they are building a large-scale cluster utilizing Training M2 chips for Anthropic's latest models, strengthening their presence in the AI research community.

5. A Dual-Track Strategy: Collaboration with NVIDIA and Proprietary Chip Development

   • By advocating for the combined use of NVIDIA GPUs and Trainium2, AWS provides customers with a wide range of options, maintaining competitiveness in both cost-performance and raw power.

6. Strengthening Market Coverage through Global Data Center Expansion

   • AWS is expanding its regions across the globe, including Latin America and Europe, while addressing region-specific needs such as the 'European Sovereign Cloud'.

Taken together, these strategies are establishing AWS as the 'infrastructure leader of the AI era.' As model sizes and inference demands continue to grow, the diverse AI services and infrastructure provided by AWS will likely become an indispensable platform for many companies and developers. In particular, Trainium2, Bedrock, and the collaboration with Anthropic serve as major incentives for choosing AWS.

Related Articles