Kimmy K2.5: The Technical Milestone of Open-Source AI as Shown by Three Scaling Dimensions
Yang Zhilin, Research Lead at Kimmy (Moonshot AI), took the stage at Nvidia GTC 2026 to deliver a lecture titled "How Kimmy K2.5 Scales." The content contains significant implications not only for AI researchers and engineers but also for investors and business professionals. Addressing the challenge of the "data wall," he revealed the process of achieving scaling across three dimensions—new optimizer, architecture, and agent design—supported by concrete figures and technical explanations.
1. Why is "3D Scaling" important?
1-1. The limits of scaling laws and the "data wall"
Traditional scaling laws are based on the idea that "model performance improves if you proportionally increase the number of training tokens, model parameters, and compute." However, there is an upper limit to the amount of high-quality training data, and the entire industry is facing what is known as the "data wall."
In response to this challenge, Yang presented three independent scaling axes: "extracting more knowledge from the same number of tokens," "enabling processing with longer contexts," and "running multiple agents in parallel."
2. First Dimension: Improving Token Efficiency—The Muon Clip Optimizer
2-1. Why improving token efficiency raises the "intelligence ceiling"
Yang stated, "Token efficiency is not just about saving compute costs; it relates to the very ceiling of AI intelligence." Assuming 50 trillion tokens of high-quality data, he explained that achieving twice the token efficiency is "effectively equivalent to training on 100 trillion tokens." In a world where data is fixed, efficiency improvements directly raise the ceiling of model performance.
2-2. The mechanism and results of the Muon Clip optimizer
Kimmy K2.5 adopts a second-order optimizer called "Muon," which replaces the Adam optimizer proposed in 2014. He reported that by using a mechanism where each gradient update is transformed to be orthogonal to the others, they achieved twice the token efficiency compared to AdamW.
However, when scaling up to 1 trillion parameters, a problem occurred where "the maximum logit values exploded to 50–100 times higher than normal, causing training to diverge." To address this, they developed a technique called "QK Clip (Query-Key Clipping)," which achieved stable training by suppressing the maximum values of the attention layer within a certain range. Yang stated, "This is the first instance in the history of machine learning where the Muon optimizer has been scaled to the 1-trillion-parameter level."
3. Second Dimension: Long-Context Processing—Kimmy Linear Architecture
3-1. Why long context is essential in the agent era
Citing comparative research between Transformers and LSTMs, Yang emphasized the characteristic that "Transformers continue to improve in accuracy as context length increases, whereas LSTMs saturate at a certain length." He stated, "For agents to execute tasks over days or weeks and handle complex work like writing a Linux kernel from scratch, a massive context is required."
3-2. The design philosophy of Kimmy Delta Attention
The core of the new "Kimmy Linear" architecture adopted by Kimmy K2.5 is a linear attention variant called "Kimmy Delta Attention."
Traditional linear attention uses a scalar (a single value) to control the memory decay rate, which meant it was essentially a binary choice between "forgetting everything" or "retaining everything." In Kimmy Delta Attention, by setting the decay rate as a diagonal matrix (different values for each channel), they designed a system where "slow-forgetting channels (long-term memory)" and "fast-updating channels (short-term memory)" can coexist.
"Kimmy Linear," which mixes full attention mechanisms (the traditional type using self-attention in all layers) and linear attention mechanisms in a 1:3 ratio, reportedly outperformed other architectures like MLA and GDN in both short-context and long-context tasks. He noted that the difference in computational efficiency is particularly significant in extremely long contexts of 1M tokens or more.
4. Third Dimension: Agent Swarms—Solving complex tasks through parallel execution
4-1. The limits of a single agent and "AI as an organization"
Mr. Yang described the design of the agent swarm as "similar to a corporate organization." An "orchestrator agent," acting as the CEO, breaks down tasks and assigns them in parallel to multiple sub-agents with different roles, such as AI researchers, frontend engineers, physics researchers, and fact-checkers. The structure involves collecting the results completed by the sub-agents and synthesizing them into a final output.
4-2. Preventing "Lazy Agents" with New Reward Design
He stated that three reward functions were introduced for training the agent swarm. The "instantiation reward" encourages the actual generation of sub-agents. The "completion reward" requires that tasks not only be generated but also completed in a meaningful way. The "result reward" is a standard metric for measuring the achievement of the overall task. Mr. Yang noted, "With only the instantiation reward, we faced the problem of agents generating a massive number of unexecutable tasks. The completion reward allowed us to suppress that issue."
By running 100 to 1,000 sub-agents in parallel, complex tasks that cannot be completed within a realistic timeframe by a single agent become achievable. Specific examples mentioned included "writing a 100-page literature review in parallel" and "executing 10 data analysis tasks simultaneously."
5. Preview of the Next-Generation Architecture: "Attention Residue"
5-1. Reinventing Residual Connections
At the end of the lecture, Mr. Yang also mentioned the new architecture "Attention Residue," which was released around the same time as Kimmy K2.5. It is an idea that generalizes the residual connection proposed in the 2015 ResNet (a mechanism that adds the output of the previous layer directly to the next layer) as an "attention mechanism" in the depth direction.
It was born from the idea, "Residual connections are LSTMs rotated by 90 degrees, so what happens if you rotate a full attention mechanism by 90 degrees?" It collects the hidden states of all previous layers, aggregates them using attention operations, and calculates the output of the current layer.
5-2. 24% Improvement in Token Efficiency
In experiments, he reported a 24% improvement in token efficiency on scaling laws. This is a calculation where "50 trillion tokens become effectively equivalent to 62 trillion tokens." He stated that consistent improvements were also seen in benchmarks focused on coding, mathematics, and reasoning (GPQA, MATH, and HumanEval).
Conclusion
What Mr. Yang's lecture demonstrates is not the simple narrative that "open source models are closing in on closed-source frontier models," but the fact that there is still significant technical room left for improving AI performance. The Adam optimizer was an invention from 2014, but Muon Clip is twice as efficient. The transformer's residual connection was an invention from 2015, but Attention Residue has improved it by 24%. The approach of "reinventing old technology" is becoming an effective answer to the modern challenge of data constraints.