The Real Reason "AI is Getting Smarter"—What is the Often-Overlooked Evolution of Algorithms?
It's not just Moore's Law! We explain the true nature of the "invisible efficiency gains" that support AI evolution in an easy-to-understand way.
Introduction | The Real Culprit Behind AI Evolution Might Be "Algorithms"
ChatGPT and Gemini are getting smarter and smarter—the reason for this is not just the "strengthening of computing power."
It is often overlooked, but actually just as important is the evolution of "algorithmic efficiency." This refers to "innovations that allow for higher performance to be extracted at a lower cost."
In this article, we will break down how algorithmic efficiency has evolved from GPT-2 to GPT-4 and toward 2027, and what kind of impact it has.
1. What is Algorithmic Efficiency? — The Secret to "Getting Smarter with the Same Power"
First, we need to understand what "algorithmic efficiency" is.
Simply put, it is "innovations that allow for smarter output using the same computing resources."
For example:
A task that previously required 100 units of computation can now be done with 10
In other words, "10x efficiency = 1 OOM improvement"
Such efficiency gains accumulate little by little every year.
2. Proof of Evolution: The Drastic Reduction in MATH Benchmark Costs
In the "MATH benchmark," where AI is tasked with solving difficult high school math problems, the "inference cost has dropped to about 1/1,000th" in just two years (= approximately 3 OOM improvement).
This is not just a "minor tweak," but an evolution born from a "total effort of structural review, training methods, and optimization techniques."
Specifically:
Data selection and augmentation
Model compression while maintaining accuracy
New training methods such as Chain of Thought
These improvements have accumulated, and as a result, "it has become possible to solve human PhD-level problems at a low cost."
3. There are two types of efficiency improvements
Algorithm evolution can be broadly categorized into the following two types.
1. Efficiency within the paradigm (evolution inside the model)
This is an improvement focused on how to achieve the same or higher performance with less computational power within the same learning framework.
Examples:
Innovations in parameter sharing
Enhancement of few-shot learning
Optimization of stochastic training
2. Paradigm expansion (unlocking applications)
This is an evolution that "makes hidden capabilities usable."
Examples:
RLHF (Reinforcement Learning from Human Feedback)
Tool integration (integration with browser operations and calculators)
Mixture of Experts (integration of specialized structures)
4. The "0.5 OOM/year" trend that has actually been ongoing
According to research institutions like Epoch AI, it has been found that in ImageNet and language modeling over the past decade, efficiency has improved at an average annual rate of 0.5 OOM (approximately 3 times).
In other words, an improvement of about 1 to 2 OOM (equivalent to a 10 to 100-fold increase in efficiency) can be expected over four years.
While the transition from GPT-2 to GPT-3 was centered on scaling up, from GPT-3 to GPT-4, and now to GPT-4o and Gemini 1.5, efficiency improvements may account for more than half of the performance gains.
5. "Algorithm evolution" is also reflected in pricing
Evidence of this evolution is actually appearing in the "price" as well.
GPT-4o offers performance equivalent to GPT-4 while being6 to 8 times cheaper (OpenAI)
Gemini 1.5 Flash offersup to 85 times the cost efficiency compared to the original GPT-4 (Google)
In other words, it is becoming easier for users to feel the "benefits" as well.
6. What will happen in 2027? — A maximum of "+3 OOM" is possible
If current trends continue, we can expectan efficiency gain of 1 to 3 OOM (up to 1,000 times) over the next four years.
Speed: Deeper reasoning in the same computation time
Lower costs: High-performance models accessible to everyone
Expansion of practical scope: Full-scale application in medicine, law, research, etc.
And most importantly, it will act as an accelerator forthe "era where AI improves AI" = recursive self-improvement.
Summary | Algorithms are causing a revolution "quietly, but surely"
While computing often gets all the attention, the efficiency evolution of algorithms is just as important, if not more so
Growth at a pace of 0.5 OOM per year over the past decade
The "invisible protagonist" supporting cost reduction, performance improvement, and social implementation
By 2027, a further efficiency improvement of +3 OOM will be within reach
If you want to know the essence of why AI is getting smarter, look at the "ingenuity inside = algorithmic evolution".