“The Real Work Starts After Training”: Behind the Scenes of Gemini: The “Global-Scale Operations” Run by Smokejumpers
Discussions about generative AI often end with “the model has become smarter.” However, what users actually interact with is not the “trained model,” but the serving infrastructure that runs 24/7 in data centers around the world. On Google’s podcast, Google AI: Release Notes, host Logan Kilpatrick interviewed Google Fellow Emanuel (Ema) Taropa, who is responsible for the behind-the-scenes work of “delivering Gemini to billions of people.”
1. The real work starts after “the model is finished. Now, let’s put it out into the world.”
The joke at the beginning of the conversation hits the core.
“Press the easy button and magic happens”—of course, reality is different. “Serving,” as referred to here, is not just about deploying a model, but keeping it running as the “connective tissue” for every product. From that moment on, the responsibility shifts from “research” to “operations.”
Ema describes it as “never a dull moment.” This is because as the quality of the model improves, the way it is used changes, and traffic and cost projections are constantly off.
2. Training vs. Serving: Determining investment allocation through an “integrated team”
The important thing is not to separate pre-training, model design, and serving. Ema says, “We don’t make trade-offs in a vacuum,” and treats
how much to invest in training
how much to optimize serving
how much customer adoption and growth is expected
as the “same decision-making process.”
Moreover, things don’t go as planned.
We eliminate the “delta between prediction and reality” every time. While improvements like 2x speed in two weeks or 4x speed in three weeks happen, they invite new demand and raise the baseline—it is truly a structure where “winning doesn’t make things easier.”
3. Caching is not a panacea: The difficulty of LLMs is that “context is dynamic”
One of the metrics developers care about is the cache hit rate. Regarding the question, “Why is it lower than other companies?”, Ema explains it in two stages.
First, while Google has “well-functioning caches” honed through Search and Spanner, LLM serving has a different structure. The targets to be cached (e.g., parts of prompts, KV cache-like things) are not calculated in the same way as before.
Second, routing. Depending on which request is sent to which server group, the resource efficiency of a single instance or the entire pool changes, and as a result, the cache hit rate fluctuates. Furthermore, products like the Gemini app, which frequently use tools and update context, are inherently less likely to have “the same input recur.”
This is the implication that “LLMs are a different type of optimization problem than web search, even if they are both distributed systems.”
4. “Where do the chips come from?”: Capacity is always a globally optimized puzzle
Every time a new model is released, a “chip shortage” occurs. Even in the conversation, when Logan says, “I feel like we’re never going to make it in time,” Ema laughs and replies, “Who said there were chips?”
The point here is the triangle of latency, capacity, and cost.
“Cut capacity here to gain latency there.”
“Make the ‘traditional parts’ other than LLMs faster and cheaper to free up resources for overall optimization.”
Having multiple “escape routes” and reconfiguring them whenever pressure points change. This is closer to market-making in a market where supply and demand shift by the second, rather than cloud operations.
5. TPU vertical integration and “organizational intensity” make the final difference
Ema even describes Google’s TPU program as the “only vertically integrated shop on Earth.”Everything from design and operations to roadmap adjustments is kept “in-house,” making it visible who needs what. This “visibility” becomes the speed of serving optimization.
And what makes this episode interesting is not just the technology, but the depiction of the people.
The metaphor of 'Smokejumpers' (firefighters who parachute into the scene of a fire ahead of time)
The 'Fire Starters' on the front end
On-call support with a 5-minute response time
The episode of being taken off duty after an incident ('That's what friends are for')
These stories illustrate the reality that the job of 'distributing' a massive model to the world ultimately depends on team culture and cohesion. From the outside, only model performance is visible, but on the inside, 'delivery grit' determines product value.
While it is certain that the Gemini models themselves are evolving (e.g., the positioning of Gemini 2.5, etc.), what determines the user experience is whether it can be delivered quickly, stably, and at a reasonable cost at all times. This episode conveys most vividly that competition in the generative AI era is shifting from 'research' to 'operations and optimization'.