SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.

Are you making a mistake while trying to improve AI Agent RAG accuracy? Why skipping normalization causes everything to fall apart

2026年4月14日 22:34

The biggest challenge when developing an AI Agent is "RAG accuracy".

In this article, I will discuss data normalization, which is unavoidable for improving RAG accuracy using the following two methods:

1. Narrowing down in bulk with ML + Embedding <Low cost>
2. Batch judgment of remaining records by LLM <Low cost>

I attempted to realize this through a "hybrid" process that doesn't incur high costs.

1. You implemented RAG but the answers are completely wrong; 90% of the cause is here

I configured RAG. I chose a vector DB as the data storage. I adjusted the chunk size. I rewrote the prompt many times.

Even so, the answers are off😿

I imagine many people have had this experience, haven't they?

When looking for the cause, it usually leads to the same place: the data before being passed to vector search is dirty.

RAG accuracy can be roughly broken down as follows.

RAG精度 = チャンク品質 × 検索精度 × LLM生成品質

The "quality" of a chunk is not just about size; the fact that entities with the same meaning exist under multiple notations is the essential problem.

In the vector data space, "Techno Bussan Co., Ltd." and "TECHNO BUSSAN" are treated as different entities.

If you input unnormalized data as is, the search hit rate drops, and the LLM cannot find the documents it should be referencing.

In this article, I will explain in a hands-on implementation style a pipeline that solves this problem by combining ML and Cohere.

2. It's useless to search in a world where there are 4 types of "Techno Bussan"

When dealing with actual enterprise data, this situation is an everyday occurrence.

# 同じ企業が4つの顔を持っている
records = [
    "テクノ物産株式会社",      # 正式名称
    "TECHNO BUSSAN Co., Ltd.", # 英語表記
    "（株）テクノ物産",         # 略称
    "テクノブッサン",           # カタカナ読み
]

What happens if this is mixed into RAG chunks?

When you ask "Tell me the sales of Techno Bussan," vector search only hits 1 or 2 out of the 4
The LLM cannot reference the information in the remaining 2-3 items and generates an incomplete answer
The user responds with the impression that "some information is missing"

This is normalization failure pattern 1: notation variation.

There are two others.

Pattern 2: Duplicate records
A state where the same document has been indexed multiple times with slightly different metadata. The same content is returned repeatedly in search results, wasting the LLM's context window.

Pattern 3: Structural inconsistency
Inconsistencies where "Sales: 12 million yen", "Sales: 12000000", and "Sales 12 million" exist as separate fields. This makes it impossible to correctly compare or aggregate numerical values.

If you try to improve RAG while leaving these three issues unaddressed, you will hit a ceiling no matter how hard you try.

3. The case where sending all records to the LLM cost $45,000 per month

"Well, why not just send everything to the LLM and normalize it!"

At first, I thought so too. In fact, LLMs are good at resolving notation variations. But if you just send everything, you will hit a realistic "wall".

# ナイーブな実装
records = load_all_records()  # 100万件

for record in records:
    prompt = f"以下を正規化してください: {record}"
    result = cohere.chat(message=prompt)  # 1件ずつ投げる
    save(result)

Cost estimate (rough):
※This is a rough estimate, so please use it only as a reference.

100万件 × 平均500トークン = 5億トークン
Cohere Command-R+: $2.5 / 1Mトークン
→ 1回の実行で $1,250
毎日実行したら月 $37,500
Claude Sonnet を使ったら月 $45,000+
※円換算156円で、￥7,020,000/月

Latency is also an issue. If processed serially, it is expected to take several days to process 1 million records.

In other words, the LLM should not be considered a "tool to send normalization requests for all records"! It should be considered a "tool to focus on cases that are difficult to judge"!

For that, what is needed first is high-speed filtering using ML (Machine Learning).

4. Making ML select "only the suspicious ones" — How to use MinHash

The concept is simple.

1 million records total
　　↓ Extract only candidate pairs for similarity using ML (fast and cheap)
Several thousand pairs
　　↓ Semantic judgment by LLM (high accuracy, high cost)
Normalized data

What you do in the ML step is narrowing down candidate pairs that say "this and this might be the same entity".

You don't need to get the correct answer; it's more like "just pick up the suspicious ones".

Detecting approximate duplicates of strings at lightning speed with MinHash / LSH

I will try to automatically detect notation variations for the company name "Techno Bussan".

I will measure similarity using a library called "MinHash".

・An algorithm for high-speed approximate calculation of Jaccard (※) similarity
※An index that expresses how "similar" two sets are on a scale of 0 to 1
　→Close to 1: Very similar
Close to 0: Almost no common elements

It is used for similarity calculation between large-scale sets.

・First, install the necessary libraries

pip install datasketch
pip install pykakasi
pip install sentence-transformers

Now, let's use the following sample code to detect this issue of notation variation.
・Execute the following. I am using a VS Code + Jupyter execution environment.


from datasketch import MinHash, MinHashLSH

def get_minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    # 文字列を3-gramに分割してシャングル化
    shingles = {text[i:i+3] for i in range(len(text) - 2)}
    for s in shingles:
        m.update(s.encode("utf-8"))
    return m

# LSHインデックスを構築（Jaccard類似度0.5以上を候補に）
lsh = MinHashLSH(threshold=0.5, num_perm=128)

records = ["テクノ物産株式会社", "テクノ物産(株)", "テクノ物産",  "テクノ物産 株式会社", "全く別の会社"]

for i, record in enumerate(records):
    lsh.insert(f"record_{i}", get_minhash(record))

# 類似候補ペアを抽出
candidate_pairs = []
for i, record in enumerate(records):
    results = lsh.query(get_minhash(record))
    for r in results:
        j = int(r.split("_")[1])
        if i < j:
            candidate_pairs.append((i, j, record, records[j]))

print(f"候補ペア数: {len(candidate_pairs)}")
# 100万件 → 数千ペアに圧縮
for pair in candidate_pairs:
    print(f"{pair[2]}  ⟷  {pair[3]}")

1) MinHash, splitting strings into 3-grams. The image is as follows.

"テクノ物産" を3-gramに分割すると：

テクノ
 クノ物
  ノ物産

→ {"テクノ", "クノ物", "ノ物産"}

Since it compares character by character, the point is that it is resistant to notation variations.
2) records: Data for actual comparison samples

When execution is complete, it will output as follows.

Here, what we found is that
the following four notation strings were determined to be similar, and

テクノ物産株式会社
テクノ物産(株)        ← 略称
テクノ物産            ← 省略形
テクノ物産 株式会社   ← スペースあり

"completely different companies" were not detected, meaning there were zero false positives. Furthermore, we found that the strength of MinHash is its ability to detect fuzzy matches, rather than just exact matches.

Through this process, we have succeeded in extracting similarities from a large group of records.

5. How to use Embedding

We will proceed with addressing the remaining issues.

Cases where the strings are completely different but the meaning is the same (e.g., "Techno Bussan" and "Techno Bussan" in Japanese) unfortunately cannot be picked up by MinHash.

Therefore, we use "Vector conversion = Embedding"! This is a method that
scores the distance between characters to measure similarity.

[Supplement to the MinHash → Embedding flow]
MinHash is strong at "notation variations within the same language," while Embedding is strong at "semantic similarity across languages."
The two are complementary, and for large-scale data, using them in the order of MinHash → Embedding is the most cost-effective approach.

Calculate semantic similarity with Cohere Embed v3

The LLM Embed we will use for conversion this time is Cohere Embed.

It is a model that converts text into high-dimensional vectors and is utilized for semantic similarity calculation, search, classification, and RAG (Retrieval-Augmented Generation) pipelines.

・First, install the necessary libraries

 pip install cohere numpy faiss-cpu scikit-learn

Let's use the following sample code to detect this issue of notation variation.
*Since MinHash is specialized for string similarity within the same language, it is unsuitable for detecting English name ↔ Japanese name pairs. This section focuses on semantic similarity detection using Embedding.

import cohere
import numpy as np
import faiss
import re
from sklearn.cluster import DBSCAN
from sklearn.metrics.pairwise import cosine_distances
from itertools import combinations

COHERE_API_KEY = "your-cohere-api-key"
co = cohere.Client(COHERE_API_KEY)

records = [
    "テクノ物産株式会社",
    "TECHNO BUSSAN",
    "テクノブッサン",
    "株式会社山田製作所",
    "ヤマダ製作所",
    "Yamada Seisakusho",
    "全く関係ない別会社ABC",
]

# =====================
# Step 1: 正規化
# =====================
def normalize_company_name(name: str) -> str:
    noise_words = [
        "株式会社", "有限会社", "合同会社", "（株）", "(株)",
        "製作所", "工業", "商事", "産業", "ホールディングス"
    ]
    result = name
    for word in noise_words:
        result = result.replace(word, "")
    return result.strip()

normalized = [normalize_company_name(r) for r in records]
print("=== 正規化後 ===")
for orig, norm in zip(records, normalized):
    print(f"  {orig} → {norm}")

# =====================
# Step 2: Embedding
# =====================
def embed_records(texts: list[str]) -> np.ndarray:
    response = co.embed(
        texts=texts,
        model="embed-multilingual-v3.0",
        input_type="search_document"
    )
    return np.array(response.embeddings, dtype="float32")

embeddings = embed_records(normalized)

# =====================
# Step 3: FAISS
# =====================
dimension = embeddings.shape[1]
index = faiss.IndexFlatIP(dimension)
faiss.normalize_L2(embeddings)
index.add(embeddings)
distances, indices = index.search(embeddings, k=5)

# =====================
# Step 4: 距離確認
# =====================
dist_matrix = cosine_distances(embeddings)
print("\n=== ペア間コサイン距離 ===")
for i in range(len(records)):
    for j in range(i+1, len(records)):
        print(f"  {records[i]:<25} ↔ {records[j]:<25}: {dist_matrix[i][j]:.4f}")

# =====================
# Step 5: DBSCAN
# =====================
clustering = DBSCAN(eps=0.15, min_samples=2, metric="cosine")
labels = clustering.fit_predict(embeddings)

# クラスタと孤立に分類
clusters = {}
noise_indices = []
for idx, label in enumerate(labels):
    if label == -1:
        noise_indices.append(idx)
    else:
        clusters.setdefault(label, []).append(records[idx])

# =====================
# Step 6: 結果出力
# =====================
print(f"\n候補クラスタ数: {len(clusters)}")
for label, group in clusters.items():
    print(f"  クラスタ {label}: {group}")

noise = [records[idx] for idx in noise_indices]
if noise:
    print(f"\n孤立レコード（LLM判定候補）: {noise}")

1) def normalize_company_name():
　　Remove common suffixes like Co., Ltd., (Inc.), etc., as noise

2) embed_records():
Convert each record (company name, address, etc.) into a 1024-dimensional vector. Text that is semantically and notationally close will also be close in vector space.

3) faiss: Narrow down "similar candidates" at high speed even with a large number of records using FAISS
・IndexFlatIP: Calculate similarity using inner product
・normalize_L2: By normalizing, inner product becomes cosine similarity
・k=5: Get the "top 5 closest" for each record

4) Clustering with DBSCAN
・eps=0.15: If cosine distance is within 0.15, judge as "close"
　*Actually, with eps=0.08, there were 0 results.
　　Value tuning is required.
・min_samples=2: Form a cluster if 2 or more items are gathered
・label=-1: "Isolated records" that do not belong to any cluster

5) Same cluster = Candidates for name resolution (master data integration) presented

When executed, you will get results like the following.

Here, the following results are obtained.

・Items identified as similar by Embedding:
→ Cluster 0: ['Techno Bussan Co., Ltd.', 'Techno Bussan'] ✅

This was matched as a candidate for "name resolution"!

・Isolated records that could not be identified ❌

孤立: ['TECHNO BUSSAN']         ← テクノ物産グループのはず
孤立: ['株式会社山田製作所']      ← 山田グループのはず
孤立: ['ヤマダ製作所']           ← 山田グループのはず
孤立: ['Yamada Seisakusho']     ← 山田グループのはず
孤立: ['全く関係ない別会社ABC']   ← 本当に孤立（正しい）

The distance between the Japanese and English names was too great to be determined by vectors.

Even with Embedding, the challenge of "not being able to determine similarity" still remains.

However, up to this point, a significant volume of data has already been normalized.

Also, as an important point, LLM vector conversion is overwhelmingly low-cost, so the fact is that the normalization process up to this point has been low-cost.

It didn't cost any money!

6. Normalizing isolated records collectively with an LLM

Unfortunately, there are still record data that cannot be determined as "the same" through vector conversion.

So, thank you for waiting! From here, the LLM, our last line of defense, makes its appearance!
It will perform similarity judgment on the "isolated record group".

Also, since we are throwing the target records in bulk for judgment this time, it is running in a super-eco mode of "using tokens only once"!

import cohere
import json

COHERE_API_KEY = "your-cohere-api-key"
co = cohere.Client(COHERE_API_KEY)

# =====================
# Input
# =====================
# ↓ 前段のEmbeddingステップの出力結果をそのまま渡す
records_clustered = ["テクノ物産株式会社", "テクノブッサン"]

records_orphans = [
    "TECHNO BUSSAN",
    "株式会社山田製作所",
    "ヤマダ製作所",
    "Yamada Seisakusho",
    "全く関係ない別会社ABC",
]

# =====================
# LLM: 全レコードをグループ化
# =====================
def group_all_records(records: list[str]) -> list[dict]:
    prompt = f"""
以下の企業名リストを同一企業ごとにグループ化してください。

入力リスト: {json.dumps(records, ensure_ascii=False)}

以下のJSON形式のみで回答してください（説明不要）:
[
  {{
    "normalized": "標準形（法人格を除去）",
    "variants": ["表記1", "表記2"],
    "confidence": 0.0〜1.0,
    "reason": "判断理由を一言で"
  }}
]

ルール:
- 言語が違っても同一企業なら同じグループにする
- 明らかに無関係なものは単独グループにする
- 法人格（株式会社、Co.,Ltd.など）は除去して標準形を決める
"""
    response = co.chat(
        model="command-r-plus-08-2024",
        message=prompt,
        temperature=0.0
    )
    raw = response.text.strip()
    raw = raw.replace("```json", "").replace("```", "").strip()
    return json.loads(raw)

# =====================
# Step 1: 全レコードをLLMでグループ化
# =====================
all_records = records_clustered + records_orphans

print("=== Step 1: 全レコードのグループ化 ===")
groups = group_all_records(all_records)
for group in groups:
    print(f"  標準形   : {group['normalized']}")
    print(f"  表記一覧 : {group['variants']}")
    print(f"  信頼度   : {group['confidence']} / 理由: {group['reason']}")

# =====================
# Step 2: 最終結果出力
# =====================
print("\n=== 最終グルーピング結果 ===")
for group in groups:
    if len(group['variants']) > 1:
        print(f"  ✅ 同一エンティティ: {group['normalized']}")
        print(f"     表記一覧 : {group['variants']}")
    else:
        print(f"  ❌ 単独    : {group['variants'][0]}")
    print()

・We are giving the LLM instructions (rules) for grouping in natural language.

ルール:
- 言語が違っても同一企業なら同じグループにする
- 明らかに無関係なものは単独グループにする
- 法人格（株式会社、Co.,Ltd.など）は除去して標準形を決める

When executed, the following judgment results from the LLM are returned.

=== Step 1: 全レコードのグループ化 ===
  標準形   : テクノ物産
  表記一覧 : ['テクノ物産株式会社', 'テクノブッサン', 'TECHNO BUSSAN']
  信頼度   : 0.9 / 理由: 類似した表記
  標準形   : 山田製作所
  表記一覧 : ['株式会社山田製作所', 'ヤマダ製作所', 'Yamada Seisakusho']
  信頼度   : 0.8 / 理由: 類似した表記
  標準形   : 全く関係ない別会社ABC
  表記一覧 : ['全く関係ない別会社ABC']
  信頼度   : 1.0 / 理由: 他の企業と無関係

=== 最終グルーピング結果 ===
  ✅ 同一エンティティ: テクノ物産
     表記一覧 : ['テクノ物産株式会社', 'テクノブッサン', 'TECHNO BUSSAN']

  ✅ 同一エンティティ: 山田製作所
     表記一覧 : ['株式会社山田製作所', 'ヤマダ製作所', 'Yamada Seisakusho']

  ❌ 単独    : 全く関係ない別会社ABC

Wow! Both English and Japanese names are accurately sorted as strings of the same entity!

With this, data normalization for RAG has been performed.

Finally, here is a supplement regarding the cost perspective of this data normalization process.

【Cost perception of this pipeline】
・MinHash: Almost 0 yen (library processing only)
・Embedding: Approx. $0.001 / 1000 items
※Actual pricing for Cohere embed-multilingual-v3.0:
Estimate assuming $0.10 / 1M tokens and an average of 10 tokens per item
・LLM judgment: Only one API call

Even for normalizing 1 million pieces of data, only "one" request to the LLM is needed. This is the essential value of a hybrid approach.

7. Summary

Including myself, haven't many of those struggling with RAG accuracy issues been working hard day and night on vector DB and prompt optimization approaches?

But if the normalization in the preceding stage is broken, no matter what you do, there was a limit to the results.

"If the input data is clean, RAG works properly"

It is a simple story, but the road to reaching that point is long.

I hope this article serves as a map for that journey.

いいなと思ったら応援しよう！

よろしければ応援お願いします！