[Distance from AI] Which words should we focus on now? | Attention

This article is easier to understand if you have read about Transformers.
If this is your first time, please start here.

First, let's start with the answer to the previous practice question.

The previous question was as follows.

Which of the following is a major difference between Transformers and the RNNs or LSTMs that came before them?
A. Learning without using weights
B. Processing the entire text at once instead of reading it in order
C. Being able to handle images and text simultaneously
D. Having humans write and provide the rules

The correct answer was B. Processing the entire text at once instead of reading it in order.

It was a story about how the world of language AI changed significantly by questioning the premise of reading in order. Thank you to everyone who participated.

🔍 How is "looking at everything at once" achieved?

Last time, I wrote that the Transformer is a structure that looks over the entire text at once.

However, even if it "looks over everything at once," it does not treat all words the same way. Within a sentence, there are words that are strongly related to the part currently being read, and words that are almost unrelated.

Attention is the mechanism that dynamically determines "which words to focus on."

👀 What is Attention?

Attention is an English word meaning to notice or focus. In other words, it is a mechanism that uses calculation to decide "which words should be focused on now" within a sentence.

In the sentence "She ate the cake. It was very sweet," to understand what "it" refers to, you need to focus on "cake." Attention automatically finds this "it -> cake" relationship from within the entire sentence.

It is natural not to understand it. When I first read it, I couldn't really visualize the idea of "calculating focus" either.

⚖️ Measuring relationships with scores

Within Attention, the "strength of the relationship" between words is expressed as a score. In other words, it is a mechanism where the degree of relationship between one word and others is quantified, and the higher the score, the more strongly that word is referenced.

If it is "She ate the cake. It was very sweet," when viewed from the word "it," the score for "cake" becomes high. The scores for "ate" and "she" become relatively low.

This score calculation is performed for all combinations of words in the sentence. As the number of words increases, the amount of calculation increases, but because it can be processed in parallel, it runs at a realistic speed.

🔀 The concept of Self-Attention

The Attention used in Transformers is specifically called Self-Attention. In short, it is a mechanism that calculates how much words within the same sentence relate to each other.

"Self" means "oneself," and it comes from the fact that instead of referencing information from the outside, it calculates relationships within the same sentence.

When I first saw the name Self-Attention, I was a bit confused, thinking, "Focusing on oneself?" It feels closer to say that "words within a sentence refer to each other."

🔗 My connection with the AI I use now

Claude, Gemini, and GPT all use this Attention to understand text.

When I consult with Claude about my writing every day, the reason it can maintain context even during long exchanges is that this Attention keeps calculating "which words are important right now." Gemini and GPT operate on the same mechanism. I feel like I've caught a glimpse of what's happening behind the scenes in those moments when I think, "It remembers so well."

From RNN to LSTM, and from Transformer to Attention. I can finally see how the technology for handling text has built up as a single flow.

Well, that's fine.

📢 Announcement

Starting next time, some articles will be for members only.

The content will be deeper and more detailed than before. There will also be more comics. I plan to deliver them on a weekly basis. Thank you for your continued support.

Practice Problem

Which of the following is the correct explanation for "Self-Attention" used in Transformers?

A. A mechanism that references external databases to supplement information
B. A mechanism where words within the same sentence calculate their relationships with each other
C. A mechanism that processes images and text simultaneously
D. A mechanism that rearranges words without using weights

The correct answer will be announced at the beginning of the next post.

Afterword

Thank you for reading this far.

GPT's image generation is incredible, isn't it?
With this, maybe even I can start to understand it better!

Following or clicking "Like" would be a great encouragement!
I look forward to your continued support!