[Distance from AI] Learning How to Forget | LSTM

First, let's start with the answer to the previous practice question.

The question from last time was as follows.

Which of the following is a correct characteristic of how an RNN processes text?
A. Looks at the entire text at once
B. Reads sequentially while carrying over previous information
C. Processes without using weights
D. Handles images and text simultaneously

The correct answer was B. Reads sequentially while carrying over previous information.

It was a structure that proceeded by passing information along like a relay. However, it had a weakness where it tended to forget the initial information as the text became longer. Thank you to everyone who participated.

🧠 How the "forgetting" problem was solved

Last time, I wrote that RNNs have a weakness where the initial information fades as the text gets longer.

This time, we are talking about LSTM, which appeared as a solution to that problem.

When I first saw the name, I learned it was an abbreviation for "Long Short-Term Memory," and I was a bit confused because it seemed to contain "long," "short," and "memory" all at once.

🔀 What is LSTM?

LSTM (Long Short-Term Memory) is translated as long-short-term memory. In other words, it is a mechanism that can store both long and short contexts by using them selectively as needed.

It keeps the RNN concept of "carrying over previous information" while adding a structure that allows it to judge for itself "what to remember and what to forget."

🚪 The mechanism of gates

The core of LSTM is a mechanism called gates. In short, they are like "valves" that control whether or not information passes through.

There are mainly three types of gates.

• Input gate: Decides how much new information to take in

• Forget gate: Decides how much old information to discard

• Output gate: Decides what to pass on to the next step

These three work together to proceed while judging "this should be remembered now" or "this can be discarded already."

I thought the name "forget gate" was interesting. The idea of intentionally designing a function to forget felt fresh.

📖 What exactly changed?

For example, suppose you have the sentence, "Ms. Suzuki is from Tokyo, studied economics in college, and now works in Osaka," and at the end, you are asked, "Where is she from?"

RNNs tended to forget information like "from Tokyo" when sentences became long. LSTM, through its forget gate, can determine that "information about place of origin is important" and retain it until the end.

🤔 Even so, there were limits

While LSTM significantly improved upon the weaknesses of RNNs, the structure of "reading sentences in order" remained unchanged.

There were still limits when sentences became extremely long or when trying to capture the relationship between words located far apart. Also, because it processes in order, it reportedly had the issue of being difficult to calculate in parallel.

I will follow up on what solutions emerged after that in the next article. When I looked it up, the name "Transformer" came up. Another name I don't know.

Oh well.

Practice Problem

Which of the following is the correct role of the "forget gate" that appears in LSTM?

A. Decides how much new information to incorporate
B. Decides how much old information to discard
C. Selects information to pass to the next step
D. Looks at the entire sentence at once

The correct answer will be announced at the beginning of the next installment.

Afterword

Thank you for reading this far.

I think the "Distance from AI" series will start to include more and more difficult elements from here on out.
I am debating whether to make articles for those who want to know more deeply as membership-exclusive content.

I would be happy if you could just prepare yourself mentally!

It would encourage me if you could "follow" or "like"!
I look forward to your continued support!