Design and Construction of AI Multi-Agents (Introduction)

Introduction

I believe many Japanese companies have been testing RAG (Retrieval-Augmented Generation) as a way to validate generative AI.
I feel that AI agents and multi-agent systems are the next trend. In preparation for this, many companies and organizations are rolling out various frameworks and services.

However, I think the industry as a whole still lacks experience with building AI agents that use ReAct or multi-agent systems, and best practices have not yet been established.
Based on my own experience and the content of research papers, the purpose of this article is to clearly summarize the current:

• Image of a multi-agent system

• Points for design and construction

and so on.

1. What is a multi-agent system?

1-1. Until now

Many LLM applications that have been verified and built so far are, in my opinion, cases where generative AI is usedonly once per taskin use cases like RAG.
(Of course, this does not apply when building more complex applications like HyDE or Advanced RAG.)

1-2. Challenges

However, with such usage methods, there are situations where LLMs cannot be utilized due to the following constraints, especially in complex tasks.

1-3. Adoption of multi-agent architecture

To solve such single-agent challenges, it is becoming more common to adopt an architecture that improves accuracy by:

• breaking down one large, complex task into multiple small, simple tasks

• making multiple LLM calls

through
We call this architecture that performsmultiple small tasks and multiple LLM calls a multi-agent system.
(Since I don't think there is a clear definition at present, I will call it that at least for the duration of this article.)

Although there are various designs, they generally involve a structure that breaks down roles and tasks into fine details. This improves overall task accuracy and avoids constraints on the number of input/output tokens.

1-4. Practical Examples of Multi-Agent Systems

The most famous and publicly documented example is, in my opinion, Sakana AI's "The AI Scientist".

This is an attempt to automate the research and development process itself using LLMs.
It achieves this by breaking down the complex and highly difficult task of "writing a research paper" into multiple tasks and processes like those mentioned above, and having LLMs execute them individually.

Reference: What is an AI Agent?

To begin with, what is an AI agent in the context of LLMs? Currently, there is no clear or common definition, but AWS defines it as follows.

While in a narrow sense, programs that use ReAct(*) to plan and utilize tools to complete assigned tasks are called agents, in this article, I will refer to programs that use LLMs to take on a single task or role as agents in a broad sense.

2. Design and Verification of Multi-Agents

2-1. Design Flow

Empirically, it seems best to follow these steps when designing agent configurations in a multi-agent system.

However, if the problem can be solved simply by adopting a design pattern, I think you should skip the task/role decomposition (step 2 above) and adopt a simpler design.
Decomposition in business operations requires a certain level of understanding of the business and its tasks, which increases operational costs.

2-2. Adoption of Design Patterns

There are agent design patterns that are often adopted to improve task accuracy using LLMs.
There is a paper that summarizes these, introducing 18 types. Here, I will introduce three representative ones.

Reflection (self-reflection) is the simplest and most effective method.
It is heavily used in The AI Scientist, and since I also introduce an implementation method using LangGraph below, I think it will be easier to get an idea of it if you try it out.

Role-based Cooperation is a design pattern that also relates to the task decomposition mentioned earlier.

2-3. Design Points

As mentioned above, there are several points to keep in mind when designing multi-agents.

1 Accuracy and required resources are a trade-off
Basically, it is said that the more you subdivide tasks and roles in a multi-agent system (i.e., the more cost you invest), the higher the accuracy becomes.You need to be aware of the response time and usage fees that can be allocated to those tasks as constraints in advance.

2 Evaluation targets
In many cases, tasks using LLMs cannot be evaluated mechanically.
Since evaluation costs in projects become high,you need to decide what to evaluate, whether it is only the final output or if the outputs of individual agents should also be properly evaluated.

3 Evaluation methods
You also need to consider evaluation methods for tasks using LLMs.
There are two methods adopted in such cases. I believe the latter will be used frequently in the future.
① Human evaluation
② LLM-as-a-Judge (Verification of the evaluation LLM is necessary)

2-4. Necessity and verification of LLM-as-a-Judge

I think there are a certain number of people who have distrust toward LLM-as-a-Judge. However, it is considered that its adoption will accelerate in the future for the cycle of high-speed agent construction, evaluation, and correction (LLMOps).
On the other hand, it is essential to verify its scoring ability rather than trusting it completely.

In advance, confirm that the results of human scoring and scoring by the evaluation LLM are consistent.

Building multi-agents involves more verification points than imagined, and it is expected that the cost of evaluating functions will be very large.
It is necessary to create a mechanism that allows for repeating evaluations of important evaluation points as quickly as possible.

2-5. AgentOps

The concept of AgentOps, which extends LLMOps targeting a single LLM function, is also emerging.
To build a highly reliable agent system,observability from the early stages of development is importantis established.

Similar to the previous discussion on evaluation, in a multi-agent system where there are many judgments by LLMs, when some strange behavior occurs,it is necessary to be able to quickly confirm where and how the mistake occurredthat is what it means.

Reference: Automation of multi-agent construction

When mechanical evaluation (including LLM-as-a-Judge) is possible, a method of having an agent system perform the design of a multi-agent system is also being researched.

In a method called ADAS, a multi-agent system that performs tasks with higher accuracy is created by repeating design -> evaluation -> saving results.

When I tried to have it design a "multi-agent for software development" as a trial, the design became as follows.

Ultimately, it is a design where agents given the roles of Tech Expert, Design Expert, Business Expert, and UX Expert repeat Reflection with each other, and the final code is output through Voting-based Cooperation.
I would appreciate it if you could check the following article for details.

3. Construction of multi-agent systems

Since I have not yet fully experienced building a production-grade multi-agent system myself, please allow me to share some simple thoughts on the matter.

Personally, I believe that multi-agent systems and Kubernetes are a good match.
This involves a configuration where each individual agent is treated as a single container, running on a container platform such as Kubernetes, OpenShift, or ECS.

In particular, once you start using ReAct, it becomes difficult to predict which agent will be called and to what extent, so it would be beneficial to have automatic scaling and health management if you are hosting your own LLM inference server.
I also think the ability to manage agent designs using YAML or similar formats is attractive.

In fact, companies like Nvidia and Red Hat seem to be moving with such a future in mind.
Nvidia provides container images for inference servers that can run LLMs at high speed through a service called NIM. They have also announced NIM Agent Blueprints, which allow you to turn multi-agent systems built by combining these into Helm charts and deploy them to Kubernetes on major clouds like AWS and Azure.

4. Major Services and Frameworks

4-1. Major Cloud Services

As of November 20, 2024, based on a quick search, AWS, Azure, and GCP are each deploying agent-related services as follows.

Although I haven't touched them in detail, the Azure AI Agent Service appeared to be a service for single agents rather than multi-agents. It seems the policy is to leave multi-agent configurations to OSS frameworks like AutoGen, which Microsoft is deploying separately.
(Please point it out if I am wrong.)

4-2. Major Frameworks

If anything, the OSS community seems to be more developed than the services provided by the cloud vendors mentioned above.

As of November 18, 2024, Dify (⭐️51.9k) had the most stars on GitHub.
AutoGen is developed by Microsoft, and Swarm is developed by OpenAI.
Also, Dify, CrewAI, and LangGraph each offer SaaS hosted by their respective developers.

5. Summary

This has become quite long, so I will summarize.

Situation:

• In Japanese companies, verification of single agents has progressed, mainly for RAG and chatbots.

• Moving forward, when leaving high-difficulty tasks to LLMs, the adoption of multi-agents will become necessary in more situations.

Multi-agent design:

• For multi-agent design, understanding the business is crucial.

• To support accuracy, it is also good to keep certain design patterns in mind.

Multi-agent construction:

• In building multi-agent systems, it is expected that verification costs will increase.

• LLMOps centered on LLM-as-a-Judge should be adopted.

That is all. I hope this helps with the flow of designing, verifying, and building multi-agent systems in Japan. If you have any discussions or thoughts, it would be helpful if you could comment or let me know on Note or X (Twitter).

X: https://twitter.com/CurveWeb

Thank you for reading.

Reference