SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.
見出し画像

Up to 40% Relevance Improvement with Complex Queries: The New Agentic Retrieval Engine

daka | Microsoft | AI

An article to deepen your understanding of Agentic Retrieval. Summary below.


What is Agentic Retrieval in Azure AI Search

The new API for agentic retrieval in Azure AI Search is designed to return more relevant results than ever before for complex queries and agent scenarios.

  • How it works

    • Define query plans and leverage conversation history and Azure OpenAI models

    • Decompose and transform complex queries, executing multiple searches simultaneously

    • Automatically integrate results and return content in a format ready for direct use by an LLM

  • Features

    • No pre- or post-processing required by the caller

    • Optimal retrieval is executed automatically

    • Can be passed directly to an LLM for response generation

  • Performance

    • Up to +33 points (+16 points on average) improvement in answer relevance for complex queries

    • Equivalent to conventional APIs for simple queries

    • Completes multiple transformations such as conversation history handling, spell correction, and paraphrasing in a single LLM call

    • High performance with the GPT-4o/4.1 family

Agentic retrieval API

Overview

This API automatically extracts and integrates the content required for answer generation from indexed documents.

  • Input

    • Query requesting content retrieval

    • Includes conversation history in conversational scenarios

  • Output

    • Formatted string ready for immediate use by an LLM

  • Main parameters

    • Only some, such as max output size

    • Identification, extraction, and integration of necessary content are fully automated

Differences from traditional search APIs

Traditional API

  • Two-layer structure of Recall (L1) + Reranking (L2)

    • L1: Retrieve candidates via text, vector, and hybrid methods

    • L2: Optimize top 50 results with a semantic ranker

  • L1 alone is insufficient. L2 significantly improves relevance

Agentic Retrieval API

  • While maintaining the two-layer structure above, query transformation processing is performed before search

  • Decomposes and transforms queries into multiple parts to retrieve more relevant documents

  • Deduplicates and integrates the retrieved documents

  • No additional processing required by the caller, pass the returned content directly to the LLM

Two new components

1. Query Planning

  • Input query -> converted into one or more search queries

  • Reflects conversation history as well

  • Paraphrasing and spell correction are also automated

  • All achieved in a single LLM call

Example

For the query "what about KB4048959 and waht systmes is it compatibel with?",

  • "What security updates are related to KB4048959?"

  • "What systems is KB4048959 compatible with?"
    , etc., are decomposed and spell-checked (paraphrases are also generated)

2. Result Merging

  • Execute L1/L2 searches for each search query

  • The obtained documents are deduplicated and integrated

  • Return the integrated results as a string
    → The caller can generate answers with an LLM without additional processing

Figure 1: Overview of search execution flow in agentic retrieval API.

Performance Benchmarks

We compared the agentic retrieval API with traditional search APIs using diverse datasets and queries across various types and use cases.

Search API Results and Agentic API Effectiveness

  • Traditional search APIs show sufficiently high relevance for "classic" search queries (general searches).

  • On the other hand, the agentic retrieval API significantly improves relevance for complex queries issued in agentic scenarios. It maintains the same level of relevance as traditional APIs for classic queries.

Key Achievements

  • Answer relevance improved by +16 points

  • Result generation rate improved by +15 points

  • Consistent improvements confirmed across over 10 domains and 6 languages

  • Improvements are particularly notable for difficult queries that require gathering information from multiple indexed documents

  • Significant improvements also seen in cases of typos or low word overlap between relevant documents and queries (+14 points in answer relevance)

Evaluation Metrics

In agentic scenarios, queries become highly complex, requiring the combination of multiple content fragments to create relevant answers. Therefore, traditional information retrieval metrics (e.g., NDCG) can only evaluate the relevance of each document individually and cannot accurately measure the integrated relevance across multiple documents required for complex queries.

Evaluation using the RAG Triad

Following existing research on RAG and agentic retrieval, we conducted an evaluation using three metrics known as the "RAG Triad":

  1. Content Relevance
    How relevant the retrieved content is to the query

  2. Answer Relevance
    How relevant the LLM-generated answer is to the query

  3. Groundedness
    How much the generated answer is based on the retrieved content, or whether it is a hallucination

We used GPT-4o for all evaluations. To measure relevance, we provided the LLM with query-text pairs and had it calculate a "relevance score" from 0 to 100. For groundedness, we also used the LLM to determine how much of the information in the generated answer was contained in the retrieved content, and rescaled this to 0-100.

Settings for a Fair Comparison

  • Both APIs issued queries with the same index and same search settings

  • The text length for answer generation was also standardized.

  • Answers were generated and evaluated using the same model and prompt.

  • In the agentic retrieval API, the API automatically calculates the string for answer generation (the maximum length can be specified by the caller; in this experiment, a maximum of 5000 tokens was used).

  • In the search API, since a sorted list of indexed documents is returned, documents from the list are concatenated into a single string until the limit is reached, and this is used for answer generation.

Evaluation Method for Each Metric

  • Content Relevance
    Prepare the input query, conversation history (if any), and the content used for answer generation (the output string for the agentic API, or the concatenated documents for the search API), have the evaluation LLM determine "how relevant this content is to the query," and calculate a score from 0 to 100.

  • Answer Relevance and Groundedness
    Generate an answer with the LLM using the retrieved content, and

    • Provide the input query and generated answer to the LLM to score "how relevant the answer is" (0-100).

    • Provide the retrieved content and generated answer to the LLM to score "how much the answer content is contained in the retrieved content" (groundedness) (0-100). Note that if the LLM returns "answer not found," groundedness is not evaluated for that query.

Datasets Used

  • Customer
    A document set provided with permission from Azure customers (hundreds of pages, requiring chunking before vectorization).

  • Support
    Hundreds of thousands of public support knowledge articles provided in multiple languages (8 languages used).

  • MIML (Multi-industry, Multi-language)
    A representative document set of 10 customer segments x 6 languages (approximately 1000 documents per segment, 60 indices in total. Traditional Chinese is excluded, only Simplified Chinese is used)

  • FDA
    English pharmaceutical document collection

  • DAYI
    Chinese pharmaceutical document collection

  • Arxiv
    Collection of scientific papers

  • MT-RAG
    Public dataset for conversational scenario verification (using all queries, conversations, and documents)

Methods for collecting and generating queries

  • Classic search queries

    • Queries actually issued by users on Bing

    • Automatically generated by giving parts of documents to an LLM (question-based, keyword-based, web search-based, concept exploration-based, etc.). Includes typos and paraphrases

  • Complex queries

Search Configuration

  • L1: Hybrid search using BM25 + OpenAI text-embedding-3-large (3072 dimensions)

  • L2: Re-ranking the top 50 results retrieved in L1 using a semantic ranker

This configuration is used commonly across both APIs, delivering high-quality search results (see the previous blog [2] for details).

Results

Content/Answer Relevance (Key Excerpts)

  • Classic Queries

    • For the MIML dataset, the conventional API score was 87.12, while the agentic API reached 87.89, an improvement of +0.76 points.

    • The Support dataset also showed a slight improvement, from 66.85 with the conventional API to 67.44 with the new API (+0.58).

  • Complex Queries

    • In the MIML dataset, the new API showed a significant improvement, reaching 57.13 compared to 46.94 for the conventional API (+11.25).

    • The Support dataset also saw an improvement of approximately +17 points, going from 43.92 to 61.06.

    • The FDA dataset was particularly notable, with a significant increase of +33 points, from 38.25 with the conventional API to 71.25 with the new API.

Key Point:
The agentic API shows particularly significant relevance improvements for complex queries.

Groundedness (Key Excerpts)

  • Classic Queries

    • In the MIML dataset, the conventional API scored '83.58' and the new API scored '83.52', showing nearly identical values (a slight difference of -0.06).

  • Complex Queries

    • For the MIML dataset, the new API improved by +0.67 points, scoring '73.49' compared to the conventional API's '72.82'.

Key Point:
Regarding grounding, there is no significant difference between the two APIs, and the new API maintains a sufficiently high level.

Answer Generation Rate

  • Classic Queries

    • In the MIML dataset, the conventional API was '90.37%' and the new API was '91.18%', a slight increase (+0.81).

  • Complex Queries

    • In MIML, it increased by about 9 points from '54.49%' to '63.96%'.

    • In the FDA dataset, there was a significant improvement of +30.5 points, from '45.58%' to '76.08%'.

Key Point:
For complex questions, the agentic API significantly increased the rate at which answers could be generated.

Metrics for cases where answers could only be generated by the agentic API for queries that the conventional API could not answer

  • Classic Query (MIML)

    • Answer Relevance: 82.85

    • Content Relevance: 67.60

    • Groundedness: 76.49

  • Complex Query (MIML)

    • Answer Relevance: 84.36

    • Content Relevance: 53.98

    • Groundedness: 70.25

Key Point:
The new API can generate answers with high relevance and groundedness even for queries that could not be answered by the conventional API.

Comparison by model (excerpt)

  • Classic Query (MIML)

    • For each model (GPT4o, GPT4o-mini, GPT4.1, GPT4.1-mini, GPT4.1-nano), the new API scores are at a consistently high level of approximately 86-88 points.

  • Complex Queries (MIML)

    • The score for the new API is 54-59 points across each model. Only GPT4.1-nano is slightly inferior, but there is no significant difference among the others.

Key Point:
Regardless of which model is used, an improvement in relevance via the agentic API can be confirmed.

Answer Relevance by Complex Query Type (Excerpt)

The agentic API improved answer relevance across various query types.

  • Aggregation

    • MIML: Conventional 68.57 → New API 72.17 (+3.60)

    • Support: Conventional 48.36 → New API 68.44 (+20.08)

  • Analytical

    • MIML: Conventional 35.88 → New API 39.38 (+3.50)

    • Support: Conventional 15.79 → New API 42.11 (+26.32)

  • Comparison

    • MIML: Previous 39.33 → New API 57.86 (+18.53)

    • Support: Previous 38.24 → New API 58.14 (+19.90)

  • Compound

    • MIML: Previous 47.61 → New API 62.56 (+14.94)

    • Support: Previous 41.86 → New API 68.02 (+26.16)

Key Point:
The new API is particularly effective for complex queries involving analysis, comparison, and compound structures.

Performance by Language (Excerpt)

The agentic API demonstrates high performance even in multilingual environments.

  • German

    • Complex Query (MIML): Previous 50.99 → New API 58.78 (+7.79)

    • Classic Query (Support): Previous 78.28 → New API 78.91 (+0.63)

  • English

    • Complex Queries (MIML): Previous 42.75 → New API 60.50 (+17.75)

    • Classic Queries (Support): Previous 77.93 → New API 78.07 (+0.14)

  • Japanese

    • Complex Queries (MIML): Previous 45.24 → New API 46.85 (+1.61)

    • Classic Queries (Support): Previous 57.33 → New API 56.89 (-0.43)

Key Takeaway:
Significant improvements were observed in complex queries for English, Chinese, German, and other languages. While the improvement in Japanese is small, the benefits of the new API can be confirmed across multiple languages overall.

How to get started with the Agentic Retrieval API

The Agentic Retrieval API is currently in public preview in select regions. For more details, please check the official documentation and other resources.

It is available in East Japan

Resources

Appendix

Types of complex queries (with examples)

  • Aggregation

    • Requests for lists, summaries, or compilations

    • Example: "What is the average response time for each region?"

  • Analytical

    • Requires deep analysis or interpretation

    • Example: "Analyze the impact of social media on mental health"

  • Comparison

    • Comparison or relationships between attributes or entities

    • Example: "Price comparison between Plan A and Plan B"

  • Compound

    • Contains multiple topics or questions

    • Example: 'Is A possible? What about in the case of B?'

  • Complex

    • Requires integration of multiple documents

    • Example: 'How to set up an Azure Search index'

  • Contextual

    • Requires understanding of background or context

    • Example: 'The significance of the Berlin Wall during the Cold War'

  • Conversational

    • Conversational or request-based format

    • Example: "How do I fix the 'Your account is blocked...' error?"

  • Exploratory

    • Requires abstract or multi-sentence explanation

    • Example: "Why should I use semantic search?"

  • Factual

    • Clear single answer

    • Example: "What is the capital of France?"

  • Filters

    • Narrowing the scope with specified conditions

    • Example: "MSFT Q2 2025 revenue"

  • Index Interrogation

    • Querying the entire index

    • Example: "What are the available policies?"

  • Misspellings

    • Handling misspellings in queries

    • Example: "How many documents were samantically r4nked?"

  • Range

    • Specified by data ranges or conditions

    • Example: "Widgets under $1000 per unit"

  • Snippet

    • Content matching a partial string in a document

    • Example: "Back your generative AI apps..."

  • Terse

    • Simple search engine query

    • Example: "Best retrieval concept queries"

いいなと思ったら応援しよう！