What is Hallucination? A Gentle Introduction to Why AI Lies and How to Spot It
When I asked an AI, it gave me the title of a book that doesn't exist.
It confidently provided a section number for a law that doesn't exist.
When you realize later that "this doesn't actually exist," the scariest part is thatthe AI looked completely unhesitating.
This phenomenon has a name.Hallucination (in Japanese, 'genkaku').
Note that the term 'lying' in the title is used for simplicity; it does not mean the AI knows it is false and is intentionally deceiving you.
It is a phenomenon where it creates incorrect information as if it were a correct statement..
In fact, in the United States in 2023, a lawyer was sanctioned after submitting multiple fictitious cases and fake citations created by ChatGPT to a court (U.S. District Court in New York; the lawyers were fined $5,000).
What the court took issue with was not the use of AI itself, butthe failure to verify the accuracy of the content before submission.
Even legal experts couldn't spot it. That is the tricky nature of this phenomenon.
By reading this article, you will understand the following three things:
What is hallucination? — In what forms does it appear?
Why is it hard for AI to say 'I don't know'? — Reasons shown by OpenAI research
How should you verify it? — 4 steps you can use starting tomorrow
This article is for non-engineers who use AI tools for work. It is not meant to make you fear AI, but to help youcontinue using it with peace of mind.
Conclusion: AI can sometimes create answers even when it is uncertain.
Hallucination is a phenomenon where AI createsplausible but incorrect information.
There is not just one cause. It involves both errors that occur during the learning phase anda system where guessing is rewarded more highly.
The solution is not to 'stop using it,' but tofocus on verifying information where errors would be problematic.
※ This article includes promotional content. If you want to learn how to interact with AI, a book for non-engineers is the shortcut.
Hallucination appears in these three forms.
First, let's learn what it looks like when it appears.
Creating fictional facts(non-existent people, institutions, or events).
Creating non-existent sources(fictional papers, books, legal precedents, or URLs).
Adding content not present in the provided materials(supplementing reasons or conclusions that were not written with natural-sounding text).
What they have in common is that they look natural as text. There are no typos, and it does not stutter. As we saw previously, since AI is simply continuing words that fit the context,
readability and factual accuracy are separate issues.
Furthermore, while answering with old information as if it were the latest is also dangerous in a professional context, this is sometimes due to the knowledge cutoff date or the age of the source material, so it is more accurate to consider these separately.
Why is it difficult to answer "I don't know"?
This is the main point. And here, there is a convincing answer.
A paper published on September 4, 2025, by researchers at OpenAI and the Georgia Institute of Technology, Why Language Models Hallucinate, compares this reason to a student facing a difficult exam.
If you leave a question you don't know blank, you are guaranteed to get it wrong. If you write a plausible answer, there is a chance you might get it right by accident. With this grading method, it is more advantageous to write something.
The evaluation of language models is the same; many grading methods have been used where a correct answer earns points, while an incorrect answer or "I don't know" earns no points. Consequently, a model that guesses is more likely to get a higher score than a model that appropriately indicates "I don't know."
In addition to this, the paper argues that statistical errors also occur at the pre-training stage.
In other words, it is not caused by a single defect.
Both the nature of learning from text and the evaluation method that favors guessing are involved.
Please remember just this one thing before you leave.
AI may create an answer instead of leaving it blank, even when it is uncertain.
While this is decreasing in newer models, it has not been completely prevented at this time.
Questions prone to this vs. usage patterns where it is relatively less likely
That said, you cannot do your job if you doubt every single answer with the same intensity. We will categorize them by risk level.
There are two axes. One is whether the AI needs to recall external facts. If you are summarizing meeting minutes you pasted, you are just rearranging the material, but asking "What is the text of this law?" forces it to try and pull information from memory.
The moment you make it recall something, you are in the danger zone.
The other is whether a mistake would affect someone's judgment or credibility. Even with the same error, the weight is different between an internal memo and a document submitted to a client. However, the risk on the right side is not zero either, as important conditions can be dropped even in summaries.
4 steps to avoid being deceived
So, how do you verify it?
First, ask for sources. Just add, "Please also provide the official information and URLs that serve as the basis." However, since AI can sometimes fabricate sources themselves, the presence of a URL does not necessarily mean the information is correct.
Therefore, next, actually open the links. Check if the page exists and if the information is truly written in the body text. It is not just about broken links; sometimes a page exists, but the content is irrelevant.
To be even more certain, ask again using the web search function. This is a method of having it refer to information obtained on the spot rather than relying solely on internal knowledge.
A similar concept is RAG (Retrieval-Augmented Generation), which creates answers after searching external documents (we will cover this in the second half of the series).
Finally, check the primary source. Even if you compare the AI's answer with a summary article, there is a possibility that they are both referencing the same misinformation.
Please trace it back to the page where it was first published whenever possible.
I use it every day myself, but I don't doubt every single character.
First, pause when numbers, dates, proper nouns, quotes, or sources appear.
Just doing this can significantly lower the risk of using serious errors as they are.
Government guidelines also require confirmation and verification
The AI Business Guidelines (Version 1.2, March 31, 2026) from the Ministry of Internal Affairs and Communications and the Ministry of Economy, Trade and Industry also addresses the risk of blindly accepting incorrect output or misinformation caused by hallucinations, and indicates that users should also adopt the mindset of verifying the accuracy of the output.
What is important is to decide on confirmation procedures that include the possibility of errors. If you are using it at a company, these are the three points.
Tasks that should not be decided solely by AI answers
Items that must be checked by a person
Official information or internal documents that should be referenced
Frequently Asked Questions
What is hallucination in simple terms? It is a phenomenon where AI creates information that sounds plausible but is incorrect. It is translated as "hallucination," but it does not have the same meaning as human hallucinations.
Is the AI lying intentionally? Usually, no. It is just that the result of creating an answer that fits the context can sometimes lead to incorrect content.
Why do hallucinations occur? In addition to statistical errors that occur during the learning phase, it is related to the fact that guessing is often more advantageous in evaluations than saying "I don't know."
Will it stop happening with new models? While there is a tendency for it to decrease, it has not reached zero at this point.
Can I prevent it by turning on web search? You can lower the risk, but you cannot prevent it completely. It can still make mistakes in how it selects or interprets search results.
What kind of questions are prone to causing it? Accurate numbers, dates, legal provisions, book or paper titles, the latest information, and information about lesser-known individuals or companies, etc.
Does it happen with summaries or translations? While it is less likely to occur, it is not zero. It may supplement explanations not found in the source material or omit important conditions.
What is RAG? It is a method of searching external documents and having the AI create an answer by referencing that content. While it can limit the basis for answers, search failures or misinterpretations can still occur.
Summary
Hallucination is a phenomenon where AI creates plausible but incorrect information. It is not necessarily an intentional lie. However, even if the text looks easy to read and confident, there is no guarantee that the content is correct. This is the main point.
And once more: AI may create an answer instead of leaving it blank, even when it is uncertain.
Therefore, checking is not a sign that you do not trust the AI. It is a standard procedure for using it after understanding how it works.
The next time you receive an answer from an AI, try looking for numbers, dates, proper nouns, citations, and sources. If you find them, open the links to verify. Those few dozen seconds will protect your credibility and that of your company.
Next time, we will cover "Cloud/SaaS," which serves as the foundation for many services, including AI.
For those who want to learn more (2 books)
If you are reading for the first time: An introductory book on the IT Passport You can organize basic terms such as generative AI, machine learning, cloud, and API in one book for non-engineers (items related to generative AI have also been added to the IT Passport exam syllabus/IPA).
If you want to go a step further: A book on AI literacy Reading a book that deals with "how much to trust AI answers" and "how to discern information" will keep your judgment at work from wavering.
*This article contains promotions (affiliate links).
Read next
What is Generative AI? A gentle introduction to the differences from LLMs, how they work, and what they can do (Part 4)
What is a prompt? A gentle introduction to how to write instructions for AI, tips, and templates (Part 3)