Integrating multiple Large Language Models (LLMs) in a single Spring Boot application has become a sought-after architectural pattern for AI-driven solutions. Let’s explore how to effectively configure multiple LLMs using Spring AI to harness the unique strengths of different language models within a single application.
1. Overview
Modern AI applications often need to leverage the strengths of different large language models (LLMs) for various reasons such as quality comparison, fallback reliability, specialized capabilities, and cost optimization. Spring AI provides a flexible architecture to integrate and orchestrate multiple LLMs with minimal effort, allowing developers to dynamically route requests based on their needs. This enables maximizing the value from diverse models while simplifying management and enhancing application robustness.
2. Configuring LLMs of Different Providers
Suppose you want to interact with both OpenAI’s GPT and Anthropic’s Claude models. Spring AI allows you to configure beans for each provider and model, disabling auto-configuration to take manual control.
To use OpenAI and Anthropic models, you first need to generate API keys from their respective platforms. For OpenAI, visit the OpenAI API keys page, log in, and create a new secret key to access their services. For Anthropic, sign up or log in at the Anthropic Developer Console to generate your API key. These keys authenticate your requests and are essential for integrating their LLMs with Spring AI.
2.1 Adding Dependencies
Below is an example of Maven dependencies you need to add to your pom.xml file to include Spring AI support for OpenAI and Anthropic APIs.
<!-- Add to your pom.xml for Maven -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-openai</artifactId>
<version>stable__jar__version</version>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-anthropic-spring-boot-starter</artifactId>
<version>stable__jar__version</version>
</dependency>
This code snippet shows two Maven dependencies for integrating Spring AI with OpenAI and Anthropic services. By adding these entries in your pom.xml, your Spring Boot application can leverage the respective AI providers’ features. Replace stable__jar__version with the actual version number you want to use.
2.2 Adding Properties
Here is an example of configuration properties to disable the chat client and set API keys for OpenAI and Anthropic services.
spring.ai.chat.client.enabled=false spring.ai.openai.api-key=YOUR_OPENAI_API_KEY spring.ai.anthropic.api-key=YOUR_ANTHROPIC_API_KEY
This configuration disables the Spring AI chat client by setting spring.ai.chat.client.enabled to false. It also specifies the API keys for OpenAI and Anthropic services, which are required for authenticating requests to these AI providers. Replace YOUR_OPENAI_API_KEY and YOUR_ANTHROPIC_API_KEY with your actual API credentials.
2.3 Bean Configuration
The following Java code demonstrates how to configure multiple ChatClient beans in a Spring Boot application to interact with different AI models from OpenAI and Anthropic.
// DemoApplication.java
package com.example.demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.anthropic.AnthropicChatModel;
import org.springframework.ai.chat.ChatClient;
@SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
@Bean
public ChatClient gpt35ChatClient() {
String apiKey = System.getenv("OPENAI_API_KEY"); // Or use @Value
return ChatClient.create(new OpenAiChatModel("gpt-3.5-turbo", apiKey));
}
@Bean
public ChatClient gpt4ChatClient() {
String apiKey = System.getenv("OPENAI_API_KEY");
return ChatClient.create(new OpenAiChatModel("gpt-4", apiKey));
}
@Bean
public ChatClient claudeSonnetChatClient() {
String apiKey = System.getenv("ANTHROPIC_API_KEY"); // Or use @Value
return ChatClient.create(new AnthropicChatModel("claude-3-sonnet-20240229", apiKey));
}
@Bean
public ChatClient claudeHaikuChatClient() {
String apiKey = System.getenv("ANTHROPIC_API_KEY");
return ChatClient.create(new AnthropicChatModel("claude-3-haiku-20240307", apiKey));
}
}
This Spring Boot application defines four different ChatClient beans, each connecting to a specific AI model from OpenAI or Anthropic. The API keys are retrieved from environment variables, allowing secure and flexible configuration. This setup enables you to use multiple AI models within the same application by injecting the appropriate ChatClient bean where needed.
2.5 Controller Example
This Java Spring Boot controller exposes a REST endpoint to interact with multiple AI chat models by selecting the model and sending a user message as a request parameter.
// LlmController.java
package com.example.demo;
import org.springframework.ai.chat.ChatClient;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.web.bind.annotation.*;
@RestController
public class LlmController {
private final ChatClient gpt35ChatClient;
private final ChatClient gpt4ChatClient;
private final ChatClient claudeSonnetChatClient;
private final ChatClient claudeHaikuChatClient;
public LlmController(
@Qualifier("gpt35ChatClient") ChatClient gpt35ChatClient,
@Qualifier("gpt4ChatClient") ChatClient gpt4ChatClient,
@Qualifier("claudeSonnetChatClient") ChatClient claudeSonnetChatClient,
@Qualifier("claudeHaikuChatClient") ChatClient claudeHaikuChatClient
) {
this.gpt35ChatClient = gpt35ChatClient;
this.gpt4ChatClient = gpt4ChatClient;
this.claudeSonnetChatClient = claudeSonnetChatClient;
this.claudeHaikuChatClient = claudeHaikuChatClient;
}
@GetMapping("/prompt")
public String ask(@RequestParam String model, @RequestParam String message) {
switch (model.toLowerCase()) {
case "gpt-3.5":
return gpt35ChatClient.prompt().user(message).call().content();
case "gpt-4":
return gpt4ChatClient.prompt().user(message).call().content();
case "claude-sonnet":
return claudeSonnetChatClient.prompt().user(message).call().content();
case "claude-haiku":
return claudeHaikuChatClient.prompt().user(message).call().content();
default:
return "Supported models: gpt-3.5, gpt-4, claude-sonnet, claude-haiku";
}
}
}
This controller class injects four different ChatClient beans, each representing a specific AI model. It exposes a GET endpoint /prompt which accepts query parameters model and message. Based on the chosen model, it routes the request to the corresponding ChatClient to send the user message and returns the AI-generated response. If the model parameter does not match any supported model, it returns a list of supported options.
2.6 Code Run and Output
The following curl commands demonstrate how to call the REST API endpoint with different AI models and messages to get responses.
curl "http://localhost:8080/prompt?model=gpt-3.5&message=Tell me a joke" curl "http://localhost:8080/prompt?model=gpt-4&message=Explain quantum computing in one sentence" curl "http://localhost:8080/prompt?model=claude-sonnet&message=Tell me about Anthropic" curl "http://localhost:8080/prompt?model=claude-haiku&message=What's the capital of France?"
Each command sends a GET request to the /prompt endpoint specifying the model and the user message as query parameters. The server processes the request using the selected AI model and returns the generated response accordingly.
3. Conclusion
Integrating multiple LLMs in Spring AI is both simple and powerful. By manually defining individual ChatClient beans, you gain flexibility to orchestrate and compare models, whether from separate providers or just different variants from one provider. This setup enhances reliability and opens up new architectural possibilities for intelligent AI-driven applications. For production-readiness, consider error handling, caching, rate limiting and dynamic model selection according to your business needs.
