Excel Analysis Showdown: 4 AIs Face Off! Surprising Differences and the 'Phantom 200-Data Incident'

Hello! This is YaroTech.

Which AI is the most accurate for Excel data analysis?

Claude, ChatGPT, Copilot, Gemini... With so many options lately, it's honestly hard to know which one to use.

So, I had 4 AIs analyze the exact same Excel file and conducted a thorough comparison.

The results... took an unexpected turn!

🖥️ Execution Environment

• AIs Used:

  • Claude Desktop (Kuro-suke)

  • ChatGPT Plus (Chappy)

  • Microsoft Copilot (Colo)

  • Google Gemini (Jimmy)

• Data: 100 entries of product names (A-E)

• Analysis Request: "Calculate the percentage of each product name"

• Test Date: October 21, 2025

🎯 What you will gain from this article

• Differences in analysis accuracy among 4 major AIs

• Characteristics of each AI and how to use them

• Insights into why results differ between AIs

• Criteria for choosing which AI to use for practical work

• The full story behind Gemini's 'Phantom 200-Data Incident'

📊 Experiment Details

Test Data

100 product data entries (No. 1-100), with 5 types of product names (A-E).
They are arranged randomly next to No. 1-100.

The actual correct answers are:

• Product E: 44 items (44.0%)

• Product D: 18 items (18.0%)

• Product B: 17 items (17.0%)

• Product A: 12 items (12.0%)

• Product C: 9 items (9.0%)

I asked each AI to perform the analysis without knowing this correct answer.

Request Prompt

添付したExcelのデータ分析をしてください。２列で下記項目名のデータです。分析としては、製品名の割合からしてください。
A列：No
B列：製品名

🤖 Analysis Results for Each AI

1. Claude Desktop (Kuro-suke) Results

Time until results: approx. 10 seconds

| Product Name | Count | Percentage (%) |
|--------|------|---------|
| E | 44 | 44.0% |
| D | 18 | 18.0% |
| B | 17 | 17.0% |
| A | 12 | 12.0% |
| C | 9 | 9.0% |

Verdict: ✅ Perfectly correct!

Features:

• Automatically generated graphs (pie chart/bar chart)

• Created 3 files:

  1. `product_analysis_chart.png` - Graph image

  2. `product_analysis_report.txt` - Detailed report

  3. `product_analysis_result.xlsx` - Aggregated Excel result

• Included analysis comment: 'Product E is overwhelmingly popular'

• Full automation including file creation

Claude's analysis comments:

2. ChatGPT Plus (Chappy) Results

Time to results: Approx. 8 seconds

| Product Name | Percentage (%) |
|--------|---------|
| E | 44.0% |
| D | 18.0% |
| B | 17.0% |
| A | 12.0% |
| C | 9.0% |

Verdict: ✅ Completely correct!

Features:

• Identical figures to Claude

• Rich in strategic insight: Risk analysis regarding the 'composition skewed toward Product E'

• Business proposal: 'Moving forward, when considering sales strategies or production plans, it would be wise to account for the dependency risk regarding demand fluctuations for Product E.'

• Output in PDF report format

• Easy to read in table format

ChatGPT's analysis comments:

3. Microsoft Copilot Results

Time to results: Approx. 15 seconds

| Product Name | Count | Percentage (%) |
|--------|------|---------|
| E | 45 items | 45% |
| D | 18 items | 18% |
| B | 18 items | 18% |
| A | 12 items | 12% |
| C | 7 items | 7% |

Verdict: ❌ Incorrect (Figures for Products E, B, and C are different)

Features:

• Numerical values differ from other AIs (E=45%, B=18%, C=7%)

• Comprehensive analysis from a business perspective

• Proposed 'cross-analysis with sales and profit margins'

• Easy to read in table format

• Ease of use within the Excel app

Copilot's analysis commentary:

Why are the numbers different? Analysis:

1. Rounding differences: Possibility of rounding 44% to 45%

2. Data reading error: Misrecognition of some data

3. Aggregation algorithm differences: Used a proprietary aggregation method

4. Google Gemini Results

Until results are announced: Multiple attempts required

1st-2nd attempts (tried with Excel, CSV)

| Product Name | Count | Percentage (%) |
|--------|------|---------|
| E | 100 items | 50.0% |
| B | 44 items | 22.0% |
| A | 23 items | 11.5% |
| D | 21 items | 10.5% |
| C | 12 items | 6.0% |

Verdict: ❌❌ Major error

Issue:

• It arbitrarily turned 100 data entries into 200!

• Claimed 'In the total of 200 data entries...'

• Hallucinated non-existent data for No. 101-200

Gemini's (incorrect) analysis comment:

3rd Attempt (Tried with Spreadsheet)

| Product Name | Count | Percentage (%) |
|--------|------|---------|
| E | 44 | 44.0% |
| B | 21 | 21.0% |
| A | 13 | 13.0% |
| D | 12 | 12.0% |
| C | 10 | 10.0% |

Verdict: ❌ Incorrect (The figures for products B, A, D, and C are wrong)

Progress:

• Correctly recognized the data count as 100

• However, misidentified Product B as 21% (the correct answer is 17%)

• Other products are also slightly off

Gemini's analysis comment:

2.5th Attempt (Pointed out errors after the 1st and 2nd attempts)

Feedback from Kenta:

Moreover, by 'customer,' it means me, right!? No, the file is definitely correct, but...

Features:

• Only recognized correctly after human intervention

• Humbly replied, 'I take your feedback seriously...'

• Still not completely accurate even after correction

🏆 Result Summary: Accuracy Ranking

| Rank | AI | Accuracy | Speed | Output |
|------|-------|--------|------|--------------|
| 🥇 1st | Claude | 100% | ⭐⭐⭐ | 3 Files |
| 🥇 1st | ChatGPT | 100% | ⭐⭐⭐ | PDF Report |
| 🥈 3rd | Copilot | 85% | ⭐⭐ | Table |
| 🥉 4th | Gemini | 0%→70% | ⭐ | Multiple Corrections |

😱 Thorough Investigation of Gemini's 'Phantom 200-Data Incident'

Why did Gemini mistakenly identify 200 items?

Hypothesis 1: File reading error

• Misrecognized the structure of the Excel file

• Counted including blank rows or formatting information

• Some internal conversion error

Hypothesis 2: Confusion with past data

• Remembering 200 items of data analyzed previously?

• Confusion with cache or training data

• Misinterpretation of the prompt

Hypothesis 3: Filling in gaps with independent reasoning

• 'Estimating' 200 records from 100 data points

• AI expanding data on its own

• Outputting while confidence is low

Why did it improve on the third attempt?

Conversion to spreadsheet is key:

• Converting the Excel file to Google Sheets

• Improved reading accuracy due to file format change

• However, it did not become completely accurate

Reaction after being pointed out

Gemini's apology:

Human-like response:

• Polite apology

• Addressing the user as 'customer'

• However, it should have read it accurately from the start

💡 Practical usage guide

🥇 Simple aggregation and verification: Copilot

Recommended Scenarios:

• When you want a quick check within Excel

• When you need business-oriented insights

• When you want to grasp general trends

Points to Note:

• Always verify numerical accuracy

• Not suitable for critical decision-making

• Cross-checking with other AIs is recommended

🥇 Detailed Analysis & Visualization: Claude

Recommended Scenarios:

• When graphs or visualizations are needed

• When you want to output in multiple file formats

• When accurate numbers are essential

Strengths:

• Automatic file generation (images, Excel, text)

• 100% accuracy

• Even more powerful with MCP integration

Related Article: Day 5: I built my own Excel automation MCP! My challenge after finding existing tools insufficient details practical examples of using Claude's Excel MCP for automation.

🥇 Strategic Insights & Proposals: ChatGPT

Recommended Scenarios：

• I want a business strategy proposal

• Risk analysis is required

• I want to share it as a PDF report

Strengths:

• Insights from a strategic perspective

• Pointing out 'dependency risks' and similar issues

• Easy-to-understand report format

😓 To avoid: Gemini (at this moment)

Issues:

• Instability in data loading

• Risk of creating phantom data

• Requires multiple rounds of corrections

The only advantage:

• Polite responses and apologies (lol)

• Slightly better if using spreadsheet format

🔧 Practice: How to use them in combination

Pattern 1: Double-check method

1. Claudeで分析（正確性重視）
2. ChatGPTで戦略考察
3. Copilotで補足確認

Benefits:

• Ensures numerical accuracy

• Gains multi-faceted perspectives

• Early risk detection

Pattern 2: Speed-focused approach

1. Copilotで素早く確認
2. 重要な場合はClaudeで検証

Pros:

• Time-saving for daily tasks

• Improve accuracy as needed

Pattern 3: Presentation preparation approach

1. Claudeでグラフ作成
2. ChatGPTで考察文章生成
3. パワポにまとめる

Pros:

• Visual material creation

• Persuasive proposals

Related article: Day 2: Graduate from PowerPoint! How I reduced training material creation from 30 minutes to 5 minutes using Marp and MCP explains in detail how to streamline presentation material creation.

⚠️ Stumbling blocks and solutions

Point 1: The numbers don't match! Which one should I trust?

Problem:
When different AIs give different results, it's hard to know which is correct.

Solution:

1. Cross-check with multiple AIs

   • Adopt results that match across two or more AIs

   • The Claude + ChatGPT combination is the strongest

2. Manual spot-checking

   • Manually count just the first 10 items

   • Compare with AI results

Point 2: Gemini produces strange results

Problem:
Creating non-existent data, numerical values do not match.

Solution:

1. Change the file format

   • Convert Excel to CSV

   • Upload to Google Sheets

2. Give up and use another AI

   • Switch early if time is valuable

Point 3: Copilot's numerical values are off

Problem:
Numerical values are slightly different, such as 45% or 18%.

Solution:

1. Use Copilot as a 'reference value'

   • Utilize for grasping general trends

   • Do not use for important decision-making

2. Go to Claude when accuracy is required

   • Check overview with Copilot → Accurate analysis with Claude

3. Use in combination with Excel functions

   • AI analysis + manual verification

🎓 Lessons learned from this experiment

1. Each AI has different strengths

Claude: Accuracy + file generation
ChatGPT: Strategic insight + reports
Copilot: Speed + convenience
Gemini: ...do your best

2. Never blindly trust AI

Lesson:

• Always cross-check with multiple AIs

• Manually verify important data

• AI is an 'auxiliary tool,' not an 'absolute answer'

3. Right tool for the right job is key

Basics of choosing the right AI:

• Speed-focused → Copilot

• Accuracy-focused → Claude

• Insight-focused → ChatGPT

• Experimental → Gemini (Promising for the future)

📊 Before/After: Changes in AI Utilization

Before (Pre-AI)

• Manual Excel aggregation: 30 minutes

• Graph creation: 20 minutes

• Report writing: 40 minutes

• Total: 90 minutes

After (Post-AI)

• Claude analysis: 10 seconds

• Automatic graph generation: 0 seconds (Created by Claude)

• ChatGPT insights: 8 seconds

• Total: 18 seconds

Reduction rate: 99.7%!

However, if you choose the wrong AI...

• Trial and error with Gemini: 15 minutes

• Copilot numerical verification: 10 minutes

• Manual re-verification: 30 minutes

• Total: 55 minutes (Reduction rate dropped to 38%)

Conclusion: Choosing the right AI is super important!

🔗 Related Articles

Please also check out past articles related to this topic:

• Day 5: I built my own Excel automation MCP! My challenge when existing tools weren't enough

• Day 3: Automating Excel tasks with VSCode x MCP! A method easier than VBA

• Day 102: I started an AI meeting by myself! Achieving a 5-role discussion with Claude Projects (10x improvement in decision-making quality)

📝 Summary

Conclusion of the 4-way AI analysis showdown

1. Accuracy: Claude = ChatGPT > Copilot > Gemini

2. Speed: ChatGPT > Claude > Copilot > Gemini

3. Practicality: Claude (file generation), ChatGPT (analysis)

4. Reliability: Cross-checking with multiple AIs is essential

Things you can practice starting today

✅ Daily aggregation: Quick check with Copilot
✅ Important analysis: Double-check with Claude + ChatGPT
✅ Presentation materials: Graphs with Claude, analysis with ChatGPT
✅ Gemini: ...Wait for a bit more growth

The most important thing

AI is convenient, but not perfect.

As Gemini's 'Phantom 200-Record Incident' taught us, AI sometimes makes surprising mistakes.

That is precisely why,

• Use multiple AIs for different tasks

• Never neglect cross-checking

• Manually verify important data

If you follow these three principles, AI will become your ultimate tool for work efficiency!

🎨 Bonus: Prompt for creating the header image

The base for today's header image was created by Chappy (ChatGPT) using the prompt below!

柔らかな照明で撮影された超現実的でプロフェッショナルな写真を作成してください。
詳細なアニメの美意識の画像を作成してください。

下記条件のnote見出し画像をサイズは横長で作成してください。サイズは必ず横長で作成してください。
## 🎨 見出し画像案

### デザインコンセプト
- **背景**: 青→紫の美しいグラデーション（データ分析のイメージ）
- **メインビジュアル**: 
  - 中央に4つのAIアイコンを円形配置（Claude、ChatGPT、Copilot、Gemini）
  - 中心にExcelファイルのアイコン（イラスト風）
  - 各AIから分析結果が飛び出すイメージ（グラフ、数値、チャート）
- **テキスト要素**:
  - 上部: 「4つのAI徹底比較」（白文字、太字）
  - 中央: 「Excel分析対決」（大きく、黄色のアクセント）
  - 下部: 「同じ依頼で見えた驚きの違い」（白文字）
- **装飾**: 
  - データの流れを示す矢印（青→白）
  - キラキラエフェクト（分析結果のイメージ）
  - 「正確性」「速度」「考察」などのラベル

### 作成手順
1. スライドサイズ（1536×1024px、横長）
2. 青→紫のグラデーション背景を設定
3. 中央に4つのAIアイコンを円形配置
4. Excelアイコンを中心に配置
5. グラフや数値のイラストを追加
6. テキストを3段構成で追加
7. 矢印とキラキラで装飾
8. YaroTechロゴを右下に12ptで控えめに配置

You can also create a header image of the same quality using this prompt!

#AI #Excel #DataAnalysis #Claude #ChatGPT #Copilot #Gemini #WorkEfficiency #GenerativeAI #YaroTech