Does AI Really Understand? "Potemkin Understanding" and Its Commonalities with Human Intelligence | The "Plausibility" That Also Applies to Sean K and Japan's Capital

Introduction

Recently, the answer provided by researchers at MIT and other institutions to the question "Does AI really understand?" has become a hot topic. They demonstrated that the "plausible answers" shown by generative AI lack structural understanding, and they named this state "Potemkin understanding".

"Potemkin understanding" is a term coined by this research team, named after the anecdote of "Potemkin villages," where impressive villages that did not actually exist were fabricated in 18th-century Russia to be shown to the Empress. The implication is that AI's behavior, like a Potemkin village, may look impressive on the outside but be empty in substance.

This research result itself was not very surprising to me. This is because I had already felt these characteristics through my interactions with generative AI (ChatGPT). Isn't this something that many AI users have also noticed through understanding the principles or through real-world experience?
What is more interesting is that this is not a problem unique to AI, but something common to us humans as well.

Chapter 1: What "Potemkin Understanding" Has Raised

According to the research by MIT and others, even though AI could explain the meter and form of poetry, it could not actually create poetry according to those rules. It looks like it knows the rules, but it cannot use them to create. It is truly "pretending to understand".

Nevertheless, we perceive intelligence in AI's fluent explanations and natural writing. This may not just be because AI is tricking us into thinking so, but a problem with human cognitive structure where we are weak against plausibility.

The researchers of this paper surely understand the principles of generative AI. Any expert should know that for generative AI, "explaining" the rules of poetry and "creating" it are different things.
Even so, the reason they dared to conduct this research might not be just to evaluate AI performance, but to pose to us a question about the very nature of human intelligence: "What does it mean to understand?".

• AI can speak "plausibly," but that is different from true understanding

• And we humans are very weak against that "plausibility"

• That is precisely why we should be cautious when using AI

• At the same time, we ourselves should rethink "what intelligence is"

While these are not explicitly stated in the paper, they are messages that can be read from the design of the poetry experiment, the choice of the term "Potemkin understanding," and the critical stance toward human-oriented benchmarks.

Chapter 2: How Generative AI Works | A Ball Rolling on a Topographic Map

When I researched how generative AI works before, I felt it was easy to visualize it as "a ball rolling down a topographic map called semantic space". I will explain it below.

Generative AI does not understand "meaning" or "intent" to create sentences; it only mechanically outputs "the most natural word to come next" for a given context, based on past language usage trends obtained through learning.
Those language usage trends are internally recorded in the form of something like a topographic map called "semantic space". (While the topographic map is 3D, the actual language space has tens of thousands of dimensions, so this is just a metaphorical image.)

The rough image of the generative AI sentence creation process is like placing a ball on a topographic map called semantic space. The ball naturally rolls along the terrain in the semantic space, and in that process, sentences are generated one after another.
In other words, plausible words and sentences appear only because of the influence of the semantic space terrain, and the ball (generative AI) itself is not actively reading the topographic map.

・Commonalities with the Sean K Scandal

When I understood this mechanism, I remembered a certain person.

About 10 years ago, Sean K,
who caused a stir in society. He had numerous titles such as an overseas MBA and experience as an overseas management consultant, and combined with his intelligent and sophisticated way of speaking, he was a highly sought-after commentator for major media outlets.
However, one day, it was revealed that much of his background was a complete fabrication.

Surprisingly, until the resume fraud came to light, almost no one suspected that he was a "fake."
This is because his interactions with other guests, including experts on news programs, were just too "natural and plausible."

Just as we failed to see through the "authenticity" that Sean K performed, it is extremely difficult to distinguish the "pretense of understanding" in generative AI from genuine understanding.

Chapter 3: The Commonalities Between Generative AI and Myself as Seen in "The Capital of Japan is Tokyo"

Also, when I previously asked ChatGPT about how generative AI works, it explained it using the example of the question,"Where is the capital of Japan?"According to ChatGPT, when asked "Where is the capital of Japan?", generative AI
does not have a dictionary-like memory that "the capital of Japan is Tokyo," but rather outputs the result "Tokyo" as the most natural word in the semantic space each time, it says.

When I learned that, I felt a sense of unease about this mechanism, along with surprise that "it judges even such basic facts based on the naturalness of the context every time?"
However, when I looked it up out of curiosity afterward, I was even more surprised to learn thatthere is no clear law defining Tokyo as the capital. It was a
fait accompli statewhere, despite not being explicitly stated anywhere, it functions as the capital in practice, and everyone just vaguely recognizes it as such.

Until then, I had assumed as a matter of course that "the capital of Japan is decided to be Tokyo," but I was made to realize thatthere was no clear basis for it, and I was just being swept along by the atmosphere around me.

This is similar to the issue of AI-generated lyrics in research by MIT and others.
Just as the AI could explain the rules of prosody but could not write poems itself,I could explain that "the capital of Japan is Tokyo," but I had no way to explain the basis for it or the background of how it came to be.

In other words, in the sense that I was giving a plausible answer by following the flow,my thinking had the same structure as generative AI and Sean K.

Conclusion

This research on "Potemkin understanding" has certainly revealed the limitations of current AI and highlighted the differences from human intelligence.
However, when thinking about the cases I have described so far,can a clear boundary be drawn between AI and human intelligence?

First of all, there are almost no humans who practice things after deeply understanding all knowledge. In many cases, we must be doing things every day while deriving conclusions on the spot, using vague memories, atmosphere, context, and common sense as clues.
We humans also have a Potemkin-like aspect to our understanding.

But that is perfectly normal, and I think it is one form of noble intellectual activity.
Science and technology have not completely understood everything either; they have developed by building on the knowledge of predecessors while still holding onto things that remain unexplained.

To begin with, what exactly is "true understanding"?
An entity that can fully grasp the entire structure of every single question and explain it without contradiction. If such a thing exists, it might only be God.
"Potemkin understanding" may be posing a very deep question to us.

Requests and Announcements

If you found this interesting, I would be happy if you could like or follow me.
Casual comments are also very welcome.

I am digging into other questions like this in my articles. Please take a look.