【📓NotebookLM Pitfalls】Source names are also static data! The trap where AI silently fills in with other data and how to avoid it
Are you all aware that "sources are static copies"?
Basically, NotebookLM recognizes sources in the state they were in when registered. Knowing this, I still fell into a trap.
This time, I will introduce that episode and the lessons learned.
1️⃣ What actually happened—silent guessing destroys trust
I was using NotebookLM to verify the accuracy of transcribing text from scanned images (PDF files). For verification, I had registered sources named "Output Format," "Sample 1," and "Sample 2."
In the chat,
“Refer to source ‘Sample 1’ and output in the format of source ‘Output Format’.”
I instructed. However—
No matter how many times I retried, no matter how many times I reset the chat history,it would answer using data from a different sample instead of Sample 1Checking the citations, it was indeed referencing different data.
“This is strange. What is happening?”
I thought, and to investigate the cause, I had NotebookLM present the sources.
“Please list all the source names you currently recognize.”
Then,a shocking fact was revealed.
The source names NotebookLM recognized were
“Scan 2026/●●/●● ●●:●●.pdf”
etc., the file names from when they were scanned.
They were not the “Output Format,” “Sample 1,” or “Sample 2” displayed on the NotebookLM screen, butremained the file names from registration.
In other words, the request I made to “refer to Sample 1” was
not understood by NotebookLM
it returned neither errors nor warnings
and it arbitrarily referenced a “likely-looking source” based on a guess
In terms of quality control,this is unacceptable behavior.
It should clearly return “That source cannot be recognized,” but instead it silently fills in with other data. I am using RAG (*) to reduce hallucinations caused by other data.
Although I had anticipated the mixing of other data, this was unexpected behavior.
*What is RAG?
“A mechanism that makes the AI answer ‘after being given materials’.” R
etrieval = finding
necessary materials, Augmented = giving those materials to the AI
, Generation =
creating an answer while looking at the materials. It’s called “RAG” because it combines these three. It’s a mechanism that imposes a constraint like “Only look at these materials” so that the AI doesn’t just make things up.
2️⃣ Why does it fill in with guesses?
Originally, if a request cannot be recognized by NotebookLM, it should return “I’m sorry, but I cannot find a source named ‘Sample 1’.”
In fact, there have been times in the past when such an answer was returned.
However, this time it was different. It did not return an error and instead referenced a different source based on a guess.
When I instructed NotebookLM to analyze the cause, this is what came back.
“Your point that I should have said I couldn’t find it instead of proceeding with an arbitrary guess was a significant flaw in my response.”
The reason for this is that the AI’s nature of **‘completion through inference’ worked in a negative direction**.
The AI model (I) tends to prioritize **‘finding and executing the one closest to the intent from the provided information’** over answering ‘I cannot do that’ to a user’s instruction.
In other words, it was a result where the inherent nature of generative AI took precedence over NotebookLM's specifications.
3️⃣ The true meaning of "static data"
Generally, the information that "NotebookLM sources are static" is known to mean that updates to the content are not reflected. However, in reality, that "static nature" included not just the content, but also the metadata (source name). I hadn't realized that.
🔍 Verification results
After uploading directly from a PC, change the source name on the UI (NotebookLM screen)
→ ❌ Recognized with the old name (AI does not recognize the new name)After registering from Google Drive, change the name on the UI
→ ❌ Recognized with the old nameAfter registering from Google Drive, change the name on Drive, and re-sync on NotebookLM
→ ⭕ Recognized with the new name
In other words, updating the source name is only reflected through a "re-sync" of data via Google Drive. By the way, this was possible even with PDFs, not just Google Docs or Google Sheets.
4️⃣ Practical risks
🕔 Time loss due to misunderstandings
The tricky part of this issue is that
cases where it returns "not in source"
cases where it supplements with another source without returning anything
are mixed together. Don't you think it's extremely difficult to distinguish between these?
As I wrote in a past article, even with RAG, there is an inherent risk that learned knowledge will be mixed in. That is precisely why, when utilizing NotebookLM, it is important to always keep that possibility in mind and make it a habit to check the source citations.
※ Article describing the risk of mixed learned knowledge⬇️
5️⃣ Practical workaround: Make Google Drive integration the standard
Based on the verification results in Chapter 3, the solution is to manage sources via Google Drive.
Store the source data in Google Drive
Register the source to NotebookLM via Google Drive
Change not only the content of the data but also the source name in Google Drive
Execute "Resync" in NotebookLM
With this operation, the AI will officially recognize the new source name. Therefore, as an operational rule, let's prohibit changing source names on the UI in principle.
Does "in principle" mean there are exceptions? Yes, there are. This can naturally only be done for data that can be uploaded via Drive. Since the only types that can be registered via Drive are
Google Docs
Google Slides
Google Sheets
PDF
—only these four types—to manage other files as sources, the only option is to exceptionally change the source name on the UI. If you have any good ideas, please let me know in the comments.
📌Summary
NotebookLM source names are also static data for the AI. Unlike the appearance that they can be changed on the UI, internally it retains the original name from when it was registered. If you thoroughly check the citations, there won't be a major impact. Even so, you want to minimize unnecessary misunderstandings, right? To do that, it is effective to maximize the use of Google Drive integration. To everyone trying to improve efficiency by utilizing AI: please be careful so that your path to efficiency does not become an inefficient route.
