SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.

Scaling Laws of Scientific Discovery: A New Paradigm for AI Research Demonstrated by ASI-ARCH

2025年7月29日 19:00

Research that takes a human researcher 2,000 hours to complete is finished by AI in just a few hours. This is not an exaggeration, but an event that actually occurred. An innovative AI system called ASI-ARCH has written a new page in human scientific research.

This AI has achieved fully autonomous research activity, where it formulates hypotheses, designs experiments, analyzes results, and generates new hypotheses without receiving instructions from anyone. The results are astonishing, with the discovery of 106 new AI technologies.

Among them were elegant solutions to problems that researchers around the world had been unable to solve for many years.

Easy-to-understand audio delivery on Spotify: "Rami's AI Tech Radio"

A survey by the World Economic Forum ranked "curiosity and lifelong learning" as the 5th most important skill needed in 2030. Curiosity is an economic necessity, not a hobby. I explain this in detail in my book, "The Ultimate Skill in the AI Era: 'Curiosity Power'."

Preface

In modern society, the capabilities of AI are growing exponentially. On the other hand, the progress of AI research itself remains bound by the linear constraints of human researchers' cognitive abilities. This is a fundamental paradox that creates a serious bottleneck in AI development.

In this human-centric development model, the speed of innovation is determined not by computational power, but by the bandwidth of human researchers' thinking and work.

To overcome this challenge, a transformative vision has been proposed. It is the concept of "Artificial Superintelligence for AI Research (ASI4AI)," where AI autonomously conducts scientific research and designs more powerful next-generation models.

Looking back at the history of science, we have gone through three major paradigms. The first is "empirical science" based on observation and experimentation. The second is "theoretical science" that describes the world with laws and equations. And the third is "computational science" that makes full use of computer simulations.

And now, we may be witnessing the dawn of the next stage, the fourth paradigm: "AI-driven scientific discovery." This suggests not only accelerating research but also the possibility of AI becoming a true collaborator in hypothesis generation and interdisciplinary knowledge integration.

The "ASI-ARCH" detailed in this paper is the very protagonist that embodies this fourth paradigm. In the discovery of neural network architectures, which is a particularly important and difficult area of AI research, ASI-ARCH has emerged as the first demonstration of AI autonomously conducting scientific research.

It is an attempt to elevate AI from a mere "tool" to a "collaborator" in the pursuit of knowledge. In this paper, we will dissect the structure of this groundbreaking system, examine its technical background and achievements, and consider its social impact.

Background: The Bottleneck of AI Research

While the performance of AI systems has improved exponentially, it is pointed out that the speed of research and development is constrained by human cognitive abilities and remains at a linear pace.

Creating the latest models requires advanced expertise and a great deal of time, and a shortage of human researchers is becoming more of a bottleneck than the abundance of computational resources. To solve this dilemma, the idea of leaving research to the AI itself is attracting attention.

For example, in a method called Neural Architecture Search (NAS), computers automatically explore model structures. However, conventional NAS only optimizes combinations of parts prepared by humans, and truly novel ideas still relied on humans.

That is where AI for AI Research appeared. This is an attempt to automate the research cycle itself, where AI autonomously formulates hypotheses, conducts experiments, analyzes results, and applies them to the next research.

In recent years, due to the dramatic improvement in the performance of LLMs, the ability of AI to write programs and understand text has increased significantly. Thanks to this, the possibility that AI can act as a researcher or engineer and contribute to scientific discovery is becoming a reality.

ASI-ARCH, born in this context, can be said to be the first system to fully demonstrate the challenge of AI designing AI models. The research team described this achievement as an AlphaGo moment in model architecture discovery. This means it was the moment when AI showed a move in model design that was so beautiful that humans had never thought of it.

Overview of Autonomous Research Systems by AI

　ASI-ARCH is an abbreviation for Artificial Superintelligence for AI Research—a grand name that, translated literally, means "Artificial Superintelligence for AI Research." In reality, it is an autonomous research platform where multiple AI agents, including LLMs, collaborate to automatically generate and evaluate new neural network architectures (structures).

　ASI-ARCH is designed to run a series of research processes without human intervention. The cycle consists primarily of three roles (modules).

Researcher
　The role responsible for proposing new model ideas and writing prototype code. While referencing past experimental data and accumulated knowledge, it devises model structures (such as network layer configurations and connection methods) with unprecedented concepts.

Engineer
　The role responsible for actually training and evaluating the proposed models. It executes code, trains the model, and measures performance. If there are any defects, it also performs debugging and corrections.

Analyst
　The role responsible for comprehensive evaluation of experimental results. It analyzes not only model performance metrics but also structural characteristics and experimental logs to derive improvements and insights for the next cycle.

　Furthermore, ASI-ARCH includes a knowledge base and a database that support these agents. The knowledge base contains important research findings in related fields, such as linear attention, which the Researcher and Analyst refer to as needed.

　The database accumulates model proposals and their results tested so far, allowing the agents to learn from past successes and failures.

　In this way, ASI-ARCH automates the proposal → experiment → analysis loop within a single integrated system. Figure 4 shows the overall configuration of ASI-ARCH.

Figure 4: Overview of our 4-module ASI-ARCH framework. This framework operates as a closed-loop evolutionary system. The cycle begins with the Researcher (purple) proposing a new architecture based on historical data. Then, the Engineer (orange-yellow) handles training and evaluation. Finally, the Analyst (blue) integrates the experimental results and enriches the findings with knowledge from the Cognition module (red). The output of this analysis directs the next evolutionary step, enabling the system to continuously improve.

Figure 4: Overview of the ASI-ARCH autonomous research framework. The research cycle proceeds in a closed loop through the collaboration of the purple "Researcher," orange "Engineer," blue "Analyst," and the red "Cognition Base," which serves as the knowledge source. Each agent sequentially performs new architecture proposals, training/evaluation, and result analysis, and by reflecting those insights into the next proposal, the AI itself continuously improves the models.

Key Points of Technical Methodology: Automated "Creation" and "Selection"

　The greatest feature of ASI-ARCH lies in the breadth of the search space and the ingenuity of the search process. While conventional NAS was an optimization of combinations of limited parts, the ASI-ARCH Researcher can freely conceive novel structures that humans might not anticipate, while still building on past data. This is because it leverages the powerful text generation and programming capabilities of LLMs to concretize "unknown ideas" as code.

　However, simply trying everything blindly does not guarantee good results. In ASI-ARCH, several strategies are employed for effective searching.

Elite-based evolutionary strategy
　Models with high performance in the database are selected as "parents," and new mutations (variations) are proposed based on them. However, diversity is also ensured by introducing randomness in the selection, not just from the top performers but from a certain range of models.

Evaluation via fitness function
　The proposed models are scored using composite evaluation metrics (fitness). Specifically, it combines two elements: (1) quantitative performance improvements such as loss during training and scores in benchmark tests, and (2) qualitative evaluation scores judged by the LLM from an expert perspective, such as the novelty, validity, and complexity of the model structure.

　It is important to incorporate evaluations close to human aesthetic judgment because relying solely on numerical values risks falling into non-essential solutions (so-called "reward hacking") by excessively optimizing the metrics.

Self-improving debugging
　In the Engineer module, if errors or anomalies are detected during model training, the system automatically analyzes the logs and provides feedback to the Researcher to correct the code. This prevents promising ideas from being discarded due to minor bugs in the code, ensuring they are nurtured to completion.

　Additionally, to streamline the search, a two-stage evaluation process has been introduced. First, a large number of candidates are roughly tested using small-scale models (e.g., with tens of millions of parameters) to select promising ones.

　Next, the selected models are scaled up to large-scale models (e.g., hundreds of millions of parameters) and verified again, enabling highly reliable evaluation without wasting computational resources.

Performance and Characteristics of New Architectures Discovered by AI

ASI-ARCH autonomously conducted 1,773 experiments over approximately 20,000 GPU hours and discovered 106 high-performance neural network architectures.

These are all linear-time attention mechanism models (high-speed variations of Transformers) that demonstrated performance exceeding conventional human-designed baselines. Table 1 shows the performance of five representative new models compared to baseline methods.

Table 1: Performance comparison of models on language modeling and commonsense reasoning benchmarks.
Comparison between human-designed baseline models (top 3 rows) and models autonomously discovered by ASI-ARCH (bottom 5 rows). Each column represents language modeling performance (lower perplexity is better) or accuracy (%) on multiple-choice questions, with the highest values in bold.

Although the models produced by ASI-ARCH are small (approximately 340 million parameters), they outperform the baselines in almost all metrics. They also clearly surpass the baselines in average performance metrics.

Table 1: Performance comparison of language models and zero-shot commonsense reasoning. The type indicates whether the model was designed by humans ( ) or discovered by AI ( ). Bold indicates the best result, and underline indicates the second-best result.

From Table 1, it can be seen that the models discovered by ASI-ARCH (indicated by the blue robot icon in the table) achieved better results across a wider range of tasks than the conventional human-designed models (pink human icon).

For example, new models such as PathGateFusionNet and ContentSharpRouter show accuracy exceeding the baselines in difficult language model reading comprehension tasks (LAMBADA) and commonsense reasoning (e.g., HellaSwag).

It is noteworthy that these performance improvements appear not just in specific tasks, but as an overall increase in average performance. In other words, it can be said that ASI-ARCH was able to create models with excellent general-purpose capabilities.

So, how did ASI-ARCH find these excellent architectures? Analysis reveals an interesting trend in design patterns. Many of the developed models featured ingeniously combined hybrid structures of methods that had been studied individually in human design.

For example, they included many complex ideas that are difficult for humans to conceive at once, such as networks that simultaneously possess "local convolution processing" and "gating mechanisms to control long-range dependencies," or structures that hierarchically combine different time-series modeling methods.

This suggests that ASI-ARCH did not merely reinvent existing knowledge, but arrived at emergent designs that fused knowledge from different fields.

Furthermore, by monitoring the exploration process of ASI-ARCH, the state of performance improvement over time has also been confirmed. Figure 6 shows how model performance changed as the exploration progressed.

Figure 6: Figure (a) plots key performance metrics against the cumulative number of evaluated samples. The average raw benchmark score of the top candidates shows a steady upward trend, as shown in Figure (b). Meanwhile, the corresponding average raw loss shows a consistent downward trend. The composite fitness score and its key components (agents, benchmarks) all show rapid initial improvement followed by a gradual plateau. The loss component of the score shows a more gradual but continuous increase throughout the process. The composite fitness score and its key components (agents, benchmarks) all show rapid initial improvement followed by a gradual plateau. The loss component of the score shows a more gradual but continuous increase throughout the process.

Figure 6: Transition of performance metrics in ASI-ARCH exploration.
(a) Transition of the average benchmark score (purple) and loss value (orange) of top candidate models. Performance improves rapidly in the initial stage and continues to improve steadily, albeit gradually, thereafter.

(b) Transition of the average value (blue) of the fitness score (composite evaluation). As learning progresses, the score plateaus, but note that this is due to the characteristics of the evaluation function that suppresses extreme values, and the actual raw data metrics (a) continue to improve as exploration progresses.

From Figure 6(a), it can be seen that as the number of exploration samples (number of models tried) increases, benchmark performance improves steadily and the loss (error) consistently decreases. While rapid improvement is seen at the beginning, followed by a transition to gradual growth, it does not plateau in the sense that the upward trend continues to the end.

On the other hand, the composite score in Figure 6(b) levels off in the latter half, but this is because the fitness evaluation is designed to saturate due to sigmoid transformation, as mentioned above.

In short, it was suggested that the more computing resources ASI-ARCH is given, the more it will steadily continue to find high-performance models. This linear relationship is also called the "scaling law of scientific discovery" by the research team.

Figure 1: A plot of the cumulative number of state-of-the-art (SOTA) architectures discovered against total compute time. The strong linear relationship indicates that the AI system's ability to discover new high-performance architectures scales effectively with the allocated computational resources.

In fact, Figure 1 plots the cumulative number of promising architectures discovered against the computational time consumed, showing a nearly proportional relationship. This implies that 'advancements in research and development can be scaled by computational resources rather than human labor,' which can be considered a groundbreaking implication.

Discussion

The greatest impact brought about by ASI-ARCH is the fact that it demonstrated that AI can autonomously engage in intellectual creative activities. The AI accomplished in a short period, using its vast computational power as a weapon, the model improvement process that humans would spend months on through trial and error.

Particularly interesting is the analysis result that, in designing the new model, insights gained from its own experiments were utilized more than past human knowledge.

This suggests that the AI devised its next ideas by placing more weight on the empirical knowledge it generated itself rather than on the literature knowledge provided by humans. It is a phenomenon that could be called the AI discovering new principles through self-study, and it signifies that the autonomy of AI research has deepened further.

However, this research has only just begun, and several challenges and points to note have been pointed out.

Massive Consumption of Computational Resources
This time, ASI-ARCH consumed as much as 20,000 GPU hours of computation time. This is an extremely large amount for resources available to universities and research institutions, and there are cost and environmental challenges to further expanding the scale of experiments. The development of more efficient search algorithms and the creation of mechanisms to share computational resources are desired.

Verification of Versatility and Reproducibility
This research is a success story in a specific field called linear attention. Whether a similar approach can be applied to other fields (such as structural design of image models or discovery of optimization algorithms) requires future verification. Also, the achievement itself is at the preprint stage before peer review, and it will become certain after independent verification and reproduction experiments by the community.

Coordination and Control with Humans
The fact that AI can autonomously advance research and development means, conversely, that models that exceed human understanding could emerge. Among the models proposed by ASI-ARCH, there may be some whose operating principles are difficult to grasp intuitively.

Human researchers have a responsibility to appropriately evaluate and select what the AI has created and to utilize it in a way that is beneficial to society. Also, the objective function design and safety of ASI-ARCH itself are important. Discussions on ethics and governance will also need to be deepened in the future to ensure that autonomous research AI is not used for malicious purposes.

Related Information and References

To understand the origins of the Transformer
Vaswani, A. et al. (2017). "Attention is All You Need." This is a monumental paper that proposed the Transformer architecture, which laid the foundation for today's generative AI.

To understand the trend of efficient architectures
Gu, A., & Dao, T. (2023). "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." A major paper on the Mamba architecture, a strong rival to the Transformer. It helps in understanding the breadth of the research area that ASI-ARCH challenged.

To deepen thoughts on self-improving AI and safety
Bostrom, N. (2014). "Superintelligence: Paths, Dangers, Strategies." A classic masterpiece that considers the long-term challenges brought about by advanced AI. Alternatively, more recent academic papers on AI alignment are also useful for deepening the ethical perspective touched upon in this article.

Afterword

AI creates an AI that will be its successor. This scenario, which was once just a fantasy, has become realistic with the advent of ASI-ARCH. Of course, this does not mean that human researchers will become unnecessary at this point.

However, it is certain that the role of researchers is changing. The parts that humans used to do through trial and error are now being handled by AI, and humans may shift to a role of indicating direction from a higher perspective and integrating ethical and social viewpoints.

The achievement of ASI-ARCH is the first step in a self-referential progress where 'AI designs further AI.' Its impact will not be limited to the technical aspect, but will also force a transformation in the way we perceive creativity and intelligence.

I hope that the ideas and knowledge gained through this article will be of some help to your business. If you felt that this article was helpful, I would be encouraged if you could 'like' or 'follow' it. I will continue to share practical know-how and the latest AI trends, so I would be happy if you continue to read.

⚠️ Important: Less than 30% of people can fully utilize ChatGPT.
Why? It is the difference in 'questioning ability'.
In fact, there is a scientific law to 'how to frame questions' that extracts 10 times the value from AI. It is a method proven by Google's 20% rule and recommended by Amazon's Bezos. Everything is revealed in 'The Ultimate Skill for the AI Era: Curiosity Power'.
Now that AI can return 'answers' instantly, the deciding factor lies in the 'questions'. Please check out my book, 'The Ultimate Skill for the AI Era: Curiosity Power', which condenses that secret and 13 practical tools you can use in your work tomorrow.

A practical guide to learning through the contrast of 'good code vs. bad code' advocated by Google Tech Lead Tom Long. From 'working code' to 'good code'. This is a must-read book that systematizes practical guidelines for writing code that is maintainable and extensible in the long term.

What is code that is not only functional but also readable, testable, and resistant to change? In this book, we explain how to write 'good code' that professional engineers practice daily, including naming conventions, function design, error handling, and principles of abstraction, with abundant examples.

What is particularly noteworthy is that it covers not only technical correctness but also perspectives on team development and code reviews. By increasing code readability, maintainability, and extensibility, you can achieve faster development speeds and reduce bugs. This is the perfect book for engineers and development leaders aiming for continuous quality improvement.

A must-read for modern people to understand the 'why' of the digital world
Legendary Princeton University professor Brian Kernighan carefully explains the foundations of computer science while avoiding technical jargon. Even without programming experience, you can understand the core of technology, from the internet, hardware, and software to AI and security.

The second edition covers the latest topics and is structured to help you acquire the digital literacy essential in modern society. From business people to students, this book is packed with knowledge to correctly understand and wisely utilize the mechanisms of the digital world.

A genuine corporate AI utilization guide born from the practices of Hitachi's 320,000 employees! 'The Textbook of Practical Generative AI' explains how to reliably achieve results in business based on over 1,000 proven case studies. With our unique 'Difficulty x Effectiveness Matrix', the optimal AI implementation order for your organization becomes clear at a glance.

Includes utilization methods that can be implemented immediately in five fields: business efficiency, system development, call centers, infrastructure maintenance, and data analysis. From prompt engineering, RAG implementation, and project management to security and compliance measures, it covers the knowledge that companies truly need. This is a practical book for the field that stands apart from theoretical books and guides for individuals.