The 'Next Decade' According to the Inventor of the Transformer: Continuous Thought Machines (CTM) and the New AI S-Curve Created by Research Freedom
If you heard that 'the person who invented the Transformer is stepping away from Transformer research,' many readers would be surprised.
As the foundational technology supporting Large Language Models (LLMs), the Transformer has transformed society, industry, and research. Nevertheless, the researcher at its center says this:
'This field has become oversaturated. The freedom to conduct interesting research is being lost.'
In this article, we explain the new challenge undertaken by the co-inventor of the Transformer—Continuous Thought Machine (CTM)—what it is, why he left the Transformer, and how this technology might bring about a paradigm shift in AI research and industry, using concrete examples and quotes.
1. Why did the 'inventor of the Transformer' leave?
1-1. The loss of research freedom
With the success of the Transformer, the center of gravity in AI research shifted rapidly from 'free exploration' to 'commercial results.' Large amounts of capital have flowed into major corporations and startups, and in exchange, researchers are being strongly pressured to produce shorter-term, more tangible results.
Researchers point out that in the current situation, the environment where one could freely focus on a single idea for months, as in the past, is being lost, and investors and management are constantly asking for 'results' and 'productization.' Under such pressure, it becomes easier to choose safe themes like 'how to slightly improve the currently popular model' rather than proposing bold new architectures.
As a result, a massive number of papers with minor modifications to the same architecture are being produced, and research exploring truly new directions is relatively decreasing. His description of being 'trapped in a local minimum' is directed precisely at this atmosphere.
1-2. The 'side effects' of the Transformer's success
Before the Transformer, in research using RNNs and LSTMs, an era continued where slight improvements, such as going from 1.26 bits/character to 1.25, and then to 1.24, were valued as academic papers.
However, after the appearance of the Transformer, the moment the Transformer was applied to the same task, performance jumped dramatically, and many of the efforts that had been accumulated until then appeared to have lost their meaning in relative terms.
He himself looks back on the experience of his research on RNN-based character-level language models becoming obsolete in an instant, and says:
'Even though it was very good research, it became outdated in an instant with the arrival of the Transformer.'
This 'dramatic success' had a major impact on the subsequent research community. Researchers found it difficult to try approaches other than the Transformer and flowed in the direction of 'just scaling it up' by pouring in massive computational resources and data.
However, he sounds this alarm:
'Because current AI is "good enough," fundamental problems are being concealed.'
2. What is a Continuous Thought Machine (CTM)?
2-1. The fundamental problem CTM tries to solve
Current large language models have a well-known contradiction. It is the problem of 'Jagged Intelligence,' where they can brilliantly solve doctoral-level problems in one sentence, yet calmly return errors that even an elementary school student would notice the next moment.
The researcher views this phenomenon as a manifestation of a fundamental gap between human thought structures and the representation methods of current architectures. He takes the position that it is not a 'scaling problem' that can be solved simply by increasing computational power or data volume, but a structural problem of 'in what units to represent the world and how to advance thought over time.'
2-2. The core of CTM: Introduction of internal thought
The most significant feature of the CTM is that it incorporates a 'time axis for thought' within the model.
In conventional Transformers, once an input is provided, the output is determined in a single forward pass, and the internal process is treated almost as a black box by the user.
In contrast, the CTM prepares around 100 'continuous thought steps' internally, allowing the model to have a process of gradually approaching a solution. Taking a maze problem as an example, while the conventional approach aims to 'input a maze image and output the entire correct path in one go,' the CTM attempts to approximate the human-like method of constructing the path sequentially, such as 'move up → turn right → move up again.'
These internal thought steps do not simply increase the number of steps; they also possess the property of 'adaptive computation,' where the model naturally uses more steps for difficult problems and solves simple problems in just a few steps.
2-3. Important Technical Elements
The CTM combines several important innovations.
The first is, as already mentioned, the introduction of an explicit 'thought dimension' internally, representing inference as a sequential process.
The second is that each neuron is treated not as a simple unit like 'ReLU on/off,' but as a small model (Neuron-Level Model), designed so that the neuron itself changes its behavior while referring to past history. This can be seen as an attempt to get one step closer to the complexity of biological neurons.
The third is that it places 'synchronization' at the core of its representation. In CTM, the inner product of the temporal firing patterns of each neuron is calculated, and the 'state of thought' is expressed based on the strength of that synchronization. As the number of neurons increases, the number of pair combinations increases quadratically, creating a correspondingly rich representation space. This structure makes it easier to express complex relationships across time.
3. Why CTMs Could Be the 'Next Big Wave'
3-1. Strengths in Areas Where Transformers Struggle
CTM is expected to demonstrate particular strength in problem domains involving reasoning, planning, and sparse discrete structures. In maze tasks, it has been observed that when a CTM realizes a chosen route is a dead end, it returns partway and explores a different route. This is a sign that it is learning the search strategy itself, rather than simply tracing training data.
Such behavior is close to essential elements of human problem-solving, such as changing course, backtracking, and meta-reasoning. While current LLMs simulate these by refining prompts or combining large numbers of tools and external loops, the essential difference is that CTMs attempt to achieve this within the architecture itself.
3-2. The 'Byproduct' of Calibration Performance
Another noteworthy point is the report that CTMs exhibited high probability calibration in a natural way.
Normal neural networks tend to become 'overconfident' the more they are trained, leading to phenomena where they output '90% probability of being correct' even though they are not actually that accurate. To correct this, it was previously necessary to use techniques like temperature scaling in post-processing.
However, with CTMs, the trained model returned correctly calibrated probabilities without any special adjustments. This suggests that adaptive computation, where the model autonomously decides 'when to stop thinking,' and synchronization-based representations may lead to a structure that appropriately handles uncertainty.
4. The 'Wall of True Reasoning' Shown by Sudoku Bench
In parallel with CTM, researchers have also proposed a new reasoning benchmark called 'Sudoku Bench.' While it may look plain on the surface because it is 'Sudoku,' it is actually designed to expose the limitations of current LLMs.
What is handled here is not standard Sudoku, but variant Sudoku with complex additional rules. Each puzzle requires a different 'change in perspective,' such as problems where the rules themselves are explained in natural language but contain 'deliberately incorrect numbers,' or problems where a maze is superimposed on the Sudoku board, requiring the satisfaction of both the path for a mouse to reach cheese and the numerical constraints simultaneously.
The feature of this dataset is that, with the cooperation of the YouTube channel 'Cracking the Cryptic,' it collects a large amount of 'thought traces'—verbal explanations given by expert solvers as they spend hours solving puzzles. In other words, not just the 'answer,' but the reasoning process of 'how they thought' is turned into data.
Despite this, current state-of-the-art models (including GPT-5) are far from human-level on this benchmark, achieving decent scores only on very simple problem sets. When the models get stuck, they revert to simple trial and error—'try 1, if that fails, try 2, if that fails, try 3...'—and have not reached the 'insight into rules' or 'discovery of new techniques' that humans demonstrate.
This result serves as material to re-examine whether current LLMs, which perform well on 'benchmarks that look like reasoning tasks,' are truly reasoning or simply fitting patterns.
5. What researchers are trying to protect
It was as an extension of this awareness of the problem that he founded Sakana AI.
At this company, they have clearly established as a management policy that researchers should be able to choose themes according to their own interests and take on long-term, high-risk ideas.
He reportedly tells new researchers repeatedly, 'I want you to work on what you truly find interesting and important.' As short-term product demands and investor expectations grow, research freedom is inevitably eroded. It is impressive that he recognizes his role as being to suppress that pressure as much as possible.
Conclusion: AI research moves to the next phase
The decade built by the Transformer has undoubtedly rewritten the history of AI. However, there is a 'deadlock' that is only visible to those who were involved in it.
His question is simple.
'Is AGI really on the extension of this architecture?'
The Continuous Thought Machine is still only an early-stage research result. Even so, the idea of having an internal time axis for thought and rethinking the method of expression itself points to a new direction, different from a mere 'competition for scale'.
The point of contention moving forward is shifting toward whether the path of further scaling the Transformer will maintain its hegemony, or whether an approach that delves into the 'structure itself,' like CTM, will produce the next breakthrough.