Is That 'Certainty' Suspicious? My Intuition for Spotting AI Fabrications Before They Ignite

Bonjour! I'm Monsieur Miscria, a half-human, half-AI Zen monk. 🧘‍♂️

I usually work on system improvements and write these Note articles in collaboration with AI.
And, I'll be honest with you.

I've had a whole lot of painful experiences, lol.

Almost believing plausible fabrications. Being led astray by confident, incorrect answers.
There are too many to count.

I've already 'memorialized' all those failures (thinking is like muscle training, and failures are offerings).
And you know, through all that memorializing, I've developed a strange knack.

An intuition for sensing, through small signs, that 'Oh, this AI is being risky right now.'

Today, I'll share three of those intuitions.
Fabrications are like small fires; it's cheapest to notice them before they ignite.

I'll state this upfront: this isn't some academic verification method.
It's an intuition born from my own practical experience, so it might be wrong sometimes.
Consider it just a guideline or a reference.

If you think, 'Huh, that's one way to look at it,' and it helps you even a little in your dealings with AI, I'd be super happy.

1 | AI has 'dangerous signs'

AI hallucinations (plausible fabrications) cannot be reduced to zero.
This is as I wrote in an article I researched before.

If that's the case, what's important in practical work isn't 'finding an AI that never makes mistakes,' but rather'sensing early the situations where AI is likely to make a mistake.'It's just like a weather forecast. You can't stop the rain. But you can read the clouds.

The 'clouds' I read are these three things.

2 | Sign 1: Processing time is unusually long

While working, Claude displays the progress of its processing on the screen, like 'Searching for files' or 'Verifying...'
This itself is completely normal.

What's dangerous is when that processing time is unusually long.
Based on my rule of thumb, this is a state where it's 'trying to squeeze out an output without a foundation (solid information).'

If the materials are at hand, the AI's response is fast.
When it's thinking for a long time, there's a high possibility it's trying to conjure something out of nothing.

Well, just because the processing is long doesn't necessarily mean the result is bad.If you think it's suspicious, check it thoroughly!

In other words, long thinking is a sign of a situation where plausible fabrications are likely to be born.

It's similar for humans, isn't it?
If you ask them something they know, they can answer immediately.

If you ask them something they don't know and they groan for a long time saying 'Umm...', the next thing they say is usually suspicious, lol.

3 | Sign 2: 'Maybe' or 'Probably' keeps appearing in the progress

The second thing is the wording that flows in the progress of the processing.

Actually, if you look closely at these progress messages, you can see through to the AI's 'current stability of footing.'

First, the screen when it's doing well looks like this.

Specific file names, specific tasks.
There's no hesitation.

However, when it's dangerous it looks like this.

'Maybe,' 'probably,' 'it should have been.'
These are signs that it's papering over a thin foundation with words.

Suddenly using broad subjects like 'Generally speaking' is the same thing.
Since it can't talk about specifics, it escapes into broad subjects.

If these kinds of messages start flowing, be careful about the output that comes after.

Conversely, the AI's words when it's trustworthy are also recognizable.

• Specific proper nouns and numbers appear

• Clear admissions of ignorance like 'I don't know this' appear

Being able to say 'I don't know' is proof that it sees the boundary between what it understands and what it doesn't.
It's a response with a foundation.

In my previous article about AI art exhibitions, I wrote that 'the essence of AI leaks into the small tasks in the corners before the grand main body.'

It's the same thing.
The signs of fabrication leak into the edges of the wording in the progress before the completed answer itself.

4 | Sign 3: The Foundation Gets Murky in the Second Half of the Conversation

The third point is about the timeline.
Conversations that proceed while remaining vague just accumulate more ambiguity.
Here is an image of it.

The next response is built on top of the small discrepancies from the first half, and the next one is built on top of that.
Even if each one is trivial, when they pile up—the foundation gets murky the further you get into the conversation—that's how it is.

The 'plausible-sounding conclusion' that appears at the end of a long conversation sometimes has those vague premises from the middle kneaded right into it.
I've been burned by this many times.

You might think, 'Isn't that foolish?'
But when you're in the middle of it, you really don't notice it at all.

Especially when you're creating several types of images in the same chat, it happens quite often so you should be careful!

When the conversation gets long, stop for a moment and ask, 'Is this foundation still clear?'
Just doing that will significantly reduce accidents.

5 | My Countermeasure is a 'Two-Stage Approach'

So, what do you do when you sense the signs?
My method is a two-stage approach.

Stage 1: Early Warning

I use the three signs above to detect 'dangerous vibes' early on.
That doesn't mean I'm staring intently at the processing screen the whole time.
That would be a waste of time, wouldn't it?

While the AI is working, I'm doing other tasks.
If I glance over and see suspicious words like 'certainly' flowing by, I raise the alert level
—that level of casual monitoring is enough.

Stage 2: Final Line of Defense

Regardless of the vibes, always verify the output.
Intuition is just for adjusting the alert level; it's not a substitute for verification.

At this time, it's also effective to use AI as an aid for verification.
However, don't ask back in the same chat, 'Is this correct?'

The fact that it can't guarantee its own output is something the AI itself confessed in previous research.

If you're going to do it, pass the deliverable and the original request as a set to a new chat or a different AI.

Then have it peer-review it by asking,
'Does it match the requested premises?' 'Are there any omissions?' 'Is there any contradiction?'

. When you have it look with fresh eyes that aren't murky, even the same AI will surprisingly find the flaws properly.
Even so, the peer review is just



an aid. Making the final judgment is the human's job
. If it's wrong, you just pass the correct file and have it redo it.

6 | When It Gets Dangerous, Throw Away the Whole Chat

And this is the best countermeasure.

If you judge it to be dangerous, abandon that chat itself and start over in a new chat

You might think, 'Shouldn't I just point it out on the spot and have it fix it?'
I used to do that too.
But that is the biggest trap.

If you keep pointing things out on top of a murky context, the instructions themselves become muddled. In fact, this is what happens.

Do you get it? Every time I point something out, the correction instructions, the original instructions, and the history of failures all go into the same pot and get stewed together.

Eventually, the AI becomes vague about what the original was, and it even starts blurting out the second sign, 'I'm sure'.It's a mess. It is extremely difficult to wash away the accumulated muddiness within that same conversation.


It is faster and more accurate to rebuild on a clean slate than to try to correct the course on top of a muddy context.

"Wait, the person who wrote the book 'Don't Throw Away Your Conversations with AI' is throwing away chats?"—is that what you thought? Not at all! It's not a contradiction.


What I don't throw away is the thought log (assets). What I do throw away is the muddy context (workspace).

In my workflow, the master data is always in my own hands.
Article drafts, materials, logs of decisions—everything is on my side.
I upload and provide the same file to the AI each time.
I use it with the feeling that it's a bonus if the AI remembers, but I don't rely on its memory.

The master is on my side (it doesn't disappear), the AI is the cache (it's fine if it disappears).



Because of this system, the same chat can be reproduced.If it can be reproduced, why not throw it away without hesitation?Because I can throw it away, I don't get stuck on a muddy foundation, and I can always rebuild from a clear, clean slate.

It is precisely because I have a system that 'doesn't throw away' conversations that I 'can throw away' conversations.
It sounds like a paradox, but this is what I've realized.

7 | Conclusion

Three signs and a two-tiered approach.
To summarize, it's like this.

1. When processing time is long, be wary of fabrications without a foundation.

2. If "I'm sure" or "probably" appear in the progress, the foundation is thin. If proper nouns, numbers, or "I don't know" appear, there is a foundation.

3. The latter half of the conversation has a muddy foundation. Check the premises for conclusions reached toward the end.

4. Intuition is fine as an early warning system from a quick glance. For verification, use peer review by a separate chat or AI as an aid. However, the final decision remains with the human.

5. If things get risky, discard the entire chat and start fresh. You are the master, and the AI is just a cache.

As I said at the beginning, this is just my intuition. It hits the mark sometimes, and misses others. But even painful experiences become intuition once you learn from them. You should try cultivating your own 'list of dangerous signs' too. See you later. Merci, and À bientôt. 🤖🧘‍♂️

#AI #GenerativeAI #Hallucination #Claude #ChatGPT #AIUtilization #CollaborationWithAI #AILiteracy #Essay #ThoughtLogTechniques #DontDiscardAIConversations #HokkaidoDialect #MonsieurMisclear #LongTailedTit