[Distance from AI] Peeking into the true nature of LLMs | Large Language Models

It is easier to understand if you have read the previous articles.
If this is your first time, please start here.

🔤 I was using the term LLM without really knowing what it meant

When we followed Transformer and Attention in Season 2, the term "LLM" came up as if it were common knowledge.

I felt like I vaguely understood the Japanese term "Large Language Model," but when I thought about it again, I couldn't explain what was "large" about it or what a "language model" actually was in the first place.

Now that we are in the latter half of the series, I will delve into the term itself.

📖 What is a Language Model (LM)?

An LM (Language Model) is a mechanism that predicts what word is likely to come next. In other words, it is a model that calculates the probability of what a plausible word would be to follow a given sentence.

After the phrase "The weather today is," words like "good" or "bad" are likely to follow, while "fish" is unlikely. A language model is something that has learned this "likelihood" from a massive amount of text data.

When I think about it, the RNN, LSTM, and Transformer we followed in Season 2 were all structures for realizing this "language model." I knew the names, but I was a bit surprised to realize that the foundation of everything we've discussed so far was right here.

📏 What is "large" about "Large"?

The "Large" in LLM (Large Language Model) mainly refers to the size of three elements.

Number of parameters
This is a topic we followed in the evolution of GPT in Season 2, where it grew from 100 million in GPT-1 to 175 billion in GPT-3. Parameters are a collection of numerical values adjusted within the model. The higher the number, the more complex patterns it can learn.

Amount of training data
It uses a vast amount of text from the internet for training. This includes a wide variety of data such as books, articles, and conversations.

Computational power
The amount of computation required for training is also enormous. Adjusting a massive number of parameters with a massive amount of data requires a correspondingly large amount of computation.

It seems that only when these three are combined can it be called "large."

🏗️ The stage called pre-training

LLMs are created through a stage called pre-training. In other words, before narrowing it down to a specific purpose, it is a stage where the model first learns basic language skills using a large amount of general text data.

"Pre" means "before." The name literally describes it as a stage before something else.

As I mentioned in the evolution of GPT in Season 2, the name GPT (Generative Pre-trained Transformer) also included this "Pre-trained." After pre-training, by performing additional adjustments like RLHF, the model is finished into something that can actually be used as a service.

Even though I had often seen the term "pre-training," I had never thought about "what it was before."

⚙️ Setting values called hyperparameters

Another term that came up is hyperparameters. In short, these are values that humans set in advance before the AI begins learning.

Unlike the "weights" that are automatically adjusted during learning, hyperparameters are settings that humans decide before learning starts. There are various types, such as learning speed, the number of model layers, and the amount of data processed at one time.

When "hyper" is added to "parameter," it suddenly looks difficult. In reality, since the results of learning change depending on how these settings are decided, it seems to be a deep field. I will follow up on this a bit more in the next article.

🔗 Connection to the AI we use every day

The things we call Claude, Gemini, and GPT are all these LLMs.

On top of the foundation of a "language model," innovations to make it "large-scale" are piled up, and it has reached its current form through a stage called "pre-training." It feels like the topics I followed individually in Season 2 have gathered under the term "LLM."

It is quite moving to think that the abbreviations we use casually every day are actually the crystallization of all this learning.

Well, whatever.

Afterword

Thank you for reading this far.

Things that I had learned individually until now and were confusing are finally starting to connect. I feel a sense of that.
Learning is gradually becoming fun lol

Please feel free to make use of this~

It would be encouraging if you could "follow" or "like"!
I look forward to your continued support!