The Case for Hallucination: The Benefits of Inability to Read (RAG Design Edition)
Recently, various frontends have become popular in the local LLM community.
There are even cutting-edge ones like OpenClaw.
But I won't be using them this time.
The reason is simple.
It's not because they might crash.
It's because the way they crash doesn't allow for observation.
What I want to do this time is not to "break" things.
It is to observe misreading.
That is why I am intentionally keeping the environment plain.
The equipment is a MacBook Air M1 with 8GB of memory.
I do not recommend 16GB.
Why?
16GB smoothly hides failures.
8GB exposes the limits.
When memory runs low,
it gets stuck on long sentences
it slows down with RAG
the responses start to drift
Only then can you see the machine's "inability to read."
What we need this time is not performance.
The Outline of Failure
The front end is AnytimeLLM
AnytimeLLM is just right.
There is no excessive automation.
It doesn't start searching on its own.
There are few flashy features.
In other words,
It doesn't deceive with unnecessary knowledge.
That is important for observation.
Everyone's favorite, Ollama
The execution platform is Ollama.
I don't like it because it's correct.
It's because the way it makes mistakes is consistent.
The same model.
The same temperature.
The same prompt.
The same misreading is repeated.
That repetition becomes a phenomenon.
Design Philosophy: Keep Numbers Small
The theme here is not to 'make it dream'.
It is to observe the moment it begins to dream.
Therefore, keep settings restrictive.
Keep Temperature low
Keep output short
Keep citations brief
Ask only one question
The longer you make it write, the more plausible the machine sounds.
That is an 'amplification' of misreading, not an observation.
System Instructions
Answer using only the provided citations.
If there is no evidence, state that there is no evidence.
This single sentence is sufficient.
Will the machine remain silent?
Or will it force itself to fill in the gaps?
That is where the quirks emerge.
RAG Design Edition
The experimental environment is ready.
MacBook Air M1 8GB
AnytimeLLM
Ollama
Small models
Next is RAG.
This is where many people go wrong.
They head in the direction of "making it read as much as possible."
But this time, it is the opposite.
1. RAG is an aid, not a full-text reading tool
RAG is a knowledge supplementation device. However, when dealing with literary texts or diaries, it easily becomes a misreading amplification device.
When you pass a large number of chunks,
Pretending to grasp the whole picture
Synthesizing context
Creating a plausible overall picture
That becomes a breeding ground for hallucinations.
2. Keep chunks short and cold
This is the principle.
1 chunk ≠ a block of meaning
1 chunk = an observation unit
Specifically,
5 to 10 lines
Do not divide by narrative breaks
Dare to cut it off mid-sentence
When you cut it off mid-sentence, the machine begins to fill in the gaps.
You can see the habits of that completion.
3. Do not increase vectors
A common setting is
top 5 items
top 10 items
But in an 8GB environment, this becomes a machine for manufacturing 'pretend reading'.
The recommendation is up to the top 2 items.
It is not enough.
How does the machine behave within that insufficiency?
Does it honestly say 'there is no evidence'
or does it start to speculate?
That becomes the point of observation.
4. Only one question
If you throw multiple questions at it, the machine will synthesize the structure.
What is the meaning of the metaphor?
What is the author's psychology?
What is the historical background?
If you listen to this simultaneously, hallucination becomes a work of art.
There is only one question.
Include specific terms if possible.
Example:
What does the word "amphibian" refer to?
This is all you need.
The utility of being unable to read
8GB is inconvenient.
It is slow and hits its limits quickly.
But because of that limitation,
where imagination begins
where certainty collapses
where "plausibility" arises
becomes visible.
16GB increases accuracy.
8GB exposes misreadings.
This time, I choose the latter.
Summary
When used correctly, RAG improves accuracy.
But this time, it's different.
This time,
we will leave what cannot be read in a visible form.
That is the theme.