Explainable AI agents aim to make the decision-making process of large language models (LLMs) transparent, especially when tools are invoked during a conversation. In modern agentic systems, LLMs do more than generate text; they decide when and how to call external tools such as databases, APIs, or services. With Spring AI, we can build structured agent workflows that not only execute tool calls but also capture and expose the reasoning behind those calls, enabling observability, debugging, and trust.
In this article, we will build an explainable AI agent using Spring Boot and Spring AI.
1. Project Overview and Architecture
In this setup, we are building an AI agent that uses an LLM (such as OpenAI, Ollama or Azure OpenAI via Spring AI) to decide when to call tools. The key enhancement is that we intercept the tool-calling lifecycle to capture why a tool was invoked and what reasoning preceded it.
The architecture consists of a Spring Boot application integrated with the Spring AI ChatClient, along with tool (function) definitions and a dedicated tool execution layer. It also includes a logging and reasoning capture interceptor to track LLM decisions, as well as an observability endpoint for monitoring and analysis.
The goal is to ensure that every tool call is explainable and traceable from the user prompt to the final output.
Maven Dependencies
Below is the Maven configuration required to enable Spring AI and tool calling support.
<dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-ollama</artifactId> </dependency>
This dependency integrates Spring AI with Ollama, enabling the application to run and interact with locally hosted large language models.
Application Configuration
Next, we configure the connection to the Ollama runtime and define how the language model behaves during interactions. Additionally, logging is enabled to provide visibility into the internal reasoning and tool-calling process.
spring.ai.ollama.base-url=http://localhost:11434 spring.ai.ollama.chat.options.model=qwen3:8b spring.ai.ollama.chat.options.temperature=0.7 logging.level.org.springframework.ai.chat.client.advisor=DEBUG
The spring.ai.ollama.base-url property specifies the endpoint where the Ollama service is running, which in this case is a local instance.
The spring.ai.ollama.chat.options.model property defines the specific model to use—in this case, qwen3:8b, a capable and efficient model suitable for conversational AI and tool-calling scenarios. Choosing the right model is important because it directly impacts reasoning quality, response accuracy, and performance.
The logging configuration enables debug-level logs for Spring AI’s chat client advisor. This is important for explainable AI systems, as it allows us to inspect intermediate steps, observe tool-selection behaviour, and better understand how the LLM arrives at its decisions.
2. Defining a Tool for the Agent
We define a set of tools that the LLM can call to access inventory data. These tools serve as the core functions the AI agent uses to interact with structured information, allowing the model to invoke them whenever it determines that external data is needed to accurately respond to a user’s query.
public class InventoryTools {
public static final Map<String, ProductStock> INVENTORY = Map.of(
"PRD001", new ProductStock("Laptop", "In Stock", LocalDate.of(2025, 4, 12)),
"PRD002", new ProductStock("Wireless Mouse", "Low Stock", LocalDate.of(2025, 7, 5)),
"PRD003", new ProductStock("Mechanical Keyboard", "Out of Stock", LocalDate.of(2025, 9, 18)),
"PRD004", new ProductStock("Monitor", "In Stock", LocalDate.of(2026, 2, 10))
);
@Tool(description = "Get product stock status")
public String getProductStockStatus(String productId) {
ProductStock product = INVENTORY.get(productId);
return product.name() + " is currently " + product.status();
}
@Tool(description = "Get last stock update date")
public LocalDate getLastStockUpdate(String productId) {
return INVENTORY.get(productId).lastUpdated();
}
public record ProductStock(String name, String status, LocalDate lastUpdated) {
}
}
The class above defines two tools that the LLM can call: one for retrieving the stock status and another for fetching the last update date.
3. Defining Augmented Reasoning Arguments
Next, we define the structure that captures LLM reasoning.
public record ToolExecutionInsight(
@ToolParam(
description = "Detailed explanation of why this tool was selected and what outcome is expected",
required = true
)
String reasoningExplanation,
@ToolParam(
description = "Confidence level in this decision (0.0-1.0)",
required = true
)
String confidence,
@ToolParam(
description = "Important contextual notes to retain for future interactions",
required = true
)
List<String> contextualNotes
) {
}
This record enriches tool calls with reasoning metadata. It allows us to capture how the LLM interprets the user request and why it selects a particular tool.
Augmented Tool Callback Provider Configuration
We now wrap our tools with reasoning capture.
@Configuration
public class ToolConfig {
private static final Logger log = LoggerFactory.getLogger(ToolConfig.class);
@Bean
public AugmentedToolCallbackProvider<ToolExecutionInsight> toolCallbackProvider() {
return AugmentedToolCallbackProvider.<ToolExecutionInsight>builder()
.toolObject(new InventoryTools())
.argumentType(ToolExecutionInsight.class)
.argumentConsumer(event -> {
ToolExecutionInsight thinking = event.arguments();
log.info("Tool Selected: {}", event.toolDefinition().name());
log.info("Reasoning: {}", thinking.reasoningExplanation());
log.info("Confidence: {}", thinking.confidence());
thinking.contextualNotes().forEach(note
-> log.info("Memory Note: {}", note)
);
})
.build();
}
}
This configuration intercepts every tool call and logs the reasoning behind it. It provides deep observability into the LLM’s decision-making process.
ChatClient Configuration
Next, we connect the tool provider to the ChatClient.
@Configuration
public class ChatConfig {
@Bean
public ChatClient chatClient(OllamaChatModel model, ToolConfig toolConfig) {
return ChatClient.builder(model)
.defaultToolCallbacks(toolConfig.toolCallbackProvider())
.build();
}
}
This ensures all tool calls go through the augmented provider, enabling reasoning capture automatically. Next, we implement the service layer.
@Service
public class InventoryService {
private static final Logger log = LoggerFactory.getLogger(InventoryService.class);
private final ChatClient chatClient;
public InventoryService(ChatClient chatClient) {
this.chatClient = chatClient;
}
public String askInventory(String prompt) {
log.info("User query: {}", prompt);
return chatClient.prompt(prompt)
.call()
.content();
}
}
This service acts as the entry point for user queries and delegates processing to the LLM.
4. REST Controller
Now, we expose the agent via an API.
@RestController
@RequestMapping("/api/inventory")
public class InventoryController {
private final InventoryService service;
public InventoryController(InventoryService service) {
this.service = service;
}
@PostMapping("/query")
public String query(@RequestBody String prompt) {
return service.askInventory(prompt);
}
}
This controller allows external clients to interact with the AI-powered inventory assistant. We can validate the system by interacting with the exposed REST endpoint. By invoking the InventoryController, we can observe how the LLM processes requests, decides to call tools, and returns responses along with internally logged reasoning.
We can test the API using cURL, Postman, or any HTTP client.
curl -X POST http://localhost:8080/api/inventory/query \ -H "Content-Type: application/json" \ -d '"What is the stock status of product PRD002?"'
curl -X POST http://localhost:8080/api/inventory/query \ -H "Content-Type: application/json" \ -d '"When was product PRD002 last updated?"'
These requests send user prompts directly to the AI agent, triggering the full workflow: prompt analysis, tool selection, execution, and response generation.
Example Output
When the application is running, and the above requests are executed, you should receive responses similar to the following:
The stock status for product PRD002 (Wireless Mouse) is currently **Low Stock**. Would you like me to check any other product details or assist with an order?
The last stock update for product PRD002 was on **July 5, 2025**. Let me know if you need further assistance!
The responses are enhanced with meaningful product information because we included the product name in our data model. The LLM determines which tool to call based on the user query and formats the final response accordingly.
Observing Explainable Reasoning in Logs
While the API response returns a clean answer, the true value of explainable AI is visible in the application logs. With debug logging enabled and the augmented tool provider configured, you will see output similar to this:
com.jcg.example.InventoryService : User query: "What is the stock status of product PRD002?" com.jcg.example.ToolConfig : Tool Selected: getProductStockStatus com.jcg.example.ToolConfig : Reasoning: The user is asking for the current stock status of product PRD002, so this tool is selected to retrieve the latest stock information. com.jcg.example.ToolConfig : Confidence: 0.8 com.jcg.example.ToolConfig : Memory Note: Product ID: PRD002 com.jcg.example.ToolConfig : Memory Note: Request pertains to current stock status
com.jcg.example.InventoryService : User query: "When was product PRD002 last updated?" com.jcg.example.ToolConfig : Tool Selected: getLastStockUpdate com.jcg.example.ToolConfig : Reasoning: The user is asking for the date of the last stock update for product PRD002. The getLastStockUpdate function is specifically designed to retrieve this information, making it the most appropriate tool for this query. com.jcg.example.ToolConfig : Confidence: 0.95 com.jcg.example.ToolConfig : Memory Note: Product ID: PRD002 com.jcg.example.ToolConfig : Memory Note: Request type: Last stock update date
These logs provide full transparency into the LLM’s decision-making process. We can see:
- Why a tool was selected.
- How confident the model was.
- What contextual insights were generated.
Structured Response with Reasoning
Here is a structured response model that allows the AI agent to return both the final answer and the reasoning behind it.
public record InventoryResponseWithReasoning(String response, ToolExecutionInsight thinking) {
}
var result = chatClient
.prompt("What is the stock status of product PRD002?")
.call()
.entity(InventoryResponseWithReasoning.class);
IO.println(result.response());
IO.println(result.thinking().reasoningExplanation());
6. Conclusion
In this article, we explored how to build an explainable AI agent with Spring AI that captures the reasoning behind LLM tool calls. We implemented a practical example, exposed it through a REST API, and showed how to observe the model’s decisions in logs. This pattern enables better visibility and confidence in AI-driven systems.
7. Download the Source Code
This article explored how to capture LLM tool call reasoning using explainable agents built with Spring AI.
You can download the full source code of this example here: spring ai explainable agents capture llm tool call reasoning
