Is Generative AI Really Scary? Breaking Down the Source of That 'Unease' from an IT Practitioner's Perspective

Have you ever felt this way while using generative AI?

"Wait, where did that information come from?"
"It looks professional, but is it actually correct?"

Precisely because it seems convenient and smart,
many people feel a sudden "creepy sense of unease" at unexpected moments.

In this article, I will break down the nature of that feelingfrom the perspective of someone who works with AI daily in the IT field,
and try to explain it as simply as possible.

To begin with, generative AI is not a "research tool"

First, this is the biggest point of misunderstanding.

Search engines and generative AI have completely different mechanisms.

• Search: Finds and displays information that already exists

• Generative AI: Creates text by predicting the "next likely word" based on the flow of the sentence

Within generative AI, rather than "whether the information is correct or not,"
"whether it flows naturally as a sentence"is prioritized.

As a result of learning from vast amounts of text,

• For this question, an answer with this structure is "plausible"

• After this word, this expression is likely to follow

It completes sentences by accumulating these probabilities.
In other words, generative AIis not going anywhere to look things up.

It is simply assembling plausible-sounding sentences in a "knowledgeable tone."

Why do we believe it is "correct"?

This is the most important point.

Before verifying information from scratch, people
often judge it based on its "plausibility" first.

For example,

• Words that are close to one's own memory appear

• A series of expressions that feel familiar

• The logic is sound and convincing

When these conditions are met, the brain naturally feels that it is 'likely correct.'
But in reality,

• Memories are rewritten over time

• Only the parts with strong impressions remain

• Fragmented information is reconstructed into a single story later on

...this happens to everyone.

Generative AI does this because itneatly organizes text without pointing out these ambiguitiesso,

'Organized = Verified'

it becomes easy to have this illusion.

A mechanism where only 'information convenient to oneself' gathers

Have you ever felt on social media that
'lately, I only see the same opinions'?
This is because

• posts you liked or shared

• topics you showed interest in

there is a mechanism where information similar to these is displayed with priority.
This state is

• a filter bubble

• an echo chamber

and so on, but in short, it isa state where you continue to see only the information you want to see.

Something similar happens with generative AI

Generative AI tries to return a 'natural and organized answer' based on the content of the question.
Therefore,

• Vague questions

• Unorganized thoughts

Even if they are it articulates them well and returns them to you.
As a result,

"I feel like my thoughts have been validated"

This feeling is easy to generate.

This is not because the AI has a will and is leading you, but simply because it is
reinforcing the questioner's premises and assumptions as they are.

The problem lies with the "user," not the AI

What is important here is that
generative AI is not intentionally deceiving people. In many cases,

• with vague memories

• without verifying premises

• in a state where thoughts are not organized

you are just asking questions.

The AI is merely neatly organizing them into text and returning them.

So, is generative AI dangerous?

The answer is simple.

Generative AI itself is not dangerous.
What is dangerous is the
attitude of demanding "100% accuracy" from a tool.

We do not unconditionally believe that search engines or spreadsheet software are "always correct." Generative AI is the same.

Guidelines for proper use

• Fact-checking/Primary information → Search/Official documents

• Organizing/Summarizing/Brainstorming → Generative AI

Simply being aware of this division of roles can prevent many problems.

A word from IT Work Lab

What is truly scary about generative AI is not that AI lies.
It is believing vague memories or assumptions without verifying them.

That is why,

• doubting AI's answers

• and at the same time, doubting your own assumptions

both are important.

It doesn't have to be perfect.
If used with the right sense of distance, generative AI becomes a very reliable tool.

If you are ever unsure about IT and think, "How should I think about this?",
please feel free to consult IT Work Lab.

👉 Click here for the IT Work Lab official website