Transformer is the core architecture behind modern AI, powering models like ChatGPT and Gemini. introduced in 2017, it revolutionized how AI processes information. The same architecture is used for training on massive datasets and for inference to generate outputs. Here we use GPT-2(small), simpler than newer ones but perfect for learning the fundamentals.
Transformer是现代人工智能的核心架构，为ChatGPT和Gemini等模型提供支持。2017年推出后，它彻底改变了人工智能处理信息的方式。相同的架构既用于在大型数据集上进行训练，也用于推理以生成输出。在这里，我们使用GPT-2(小型版)，它比更新的模型更简单，但非常适合学习基础知识。

2. How Transformers Work? Transformer 是如何工作的?

Transformers aren't magic— they build text step by step by asking:
Transformer 并非魔法——它通过逐步提问来构建文本：
"What is the most probable next word that will follow this input?"
“接下来最有可能跟随这个输入的词是什么?”
Here we explore how a trained model generates text. Write your own text or use an example, then click generate to see it in action. If the model isn't ready yet, try another Example.
在这里，我们将探讨训练好的模型如何生成文本。输入你自己的文本或使用示例，然后点击生成按钮查看效果。如果模型尚未准备好，请尝试另一个示例。

3. Transformer Architecture Transformer架构

Transformer has three main parts:
Transformer有三个主要部分：
1嵌dings turn text into numbers.
嵌入将文本转换为数字。
②Transformer blocks mix information with Self-Attention and refine it with an MLP.
Transformer模块通过自我注意混合信息，并通过MLP对其进行细化。
③Probabilities determine the likelihood of each next token.
概率决定了每个下一个标记的可能性。

4. embedding 嵌入

Before a Transformer can use text, it first breaks it into small units and represents each as a list of numbers (vector). This process is called embedding, and the term can refer to both the process and the resulting vector.
在 Transformer 使用文本之前，它首先将其分解为小单元，并将每个单元表示为数字列表(向量)。这个过程称为嵌入，该术语既可指代过程，也可指代结果向量。
In this tool, each vector appears as a rectangle, and hover ing over it shows its size.
在这个工具中，每个向量以矩形的形式出现，悬停在其上会显示其大小。

5. token embedding 词嵌入

Tokenization splits input text into tokens— small units like words or parts of words. GPT-2(small) has 50,257 token vocabulary, each with a unique ID.
分词将输入文本分割成词元(token)，词元是像单词或单词片段这样的小单元。GPT-2(小型版)拥有50，257个词元词汇表，每个词元都有一个唯一的ID。
In the token embedding step, every token is matched to a 768-number vector from a large lookup table. These vectors are learned during training to best represent each token's meaning.
在词嵌入步骤中，每个词元都与一个来自大型查找表的768位数字向量进行匹配。这些向量在训练过程中学习，以最佳方式表示每个词元的含义。

6.PositionalEncoding 位置编码

Word order matters in language. Positional encoding gives each token information about its place in the sequence.
词序在语言中很重要。位置编码为每个标记提供有关其在序列中位置的信息。
GPT-2 does this by adding a learned positional embedding to the token's embedding, but newer models may use other methods, like RoPE, which encodes position by rotating certain vectors. All aim to help the model understand order in text.
GPT-2通过将学习到的位置嵌入添加到标记的嵌入中来实现这一点，但更新的模型可能使用其他方法，如RoPE，它通过旋转某些向量来编码位置。所有这些方法都旨在帮助模型理解文本中的顺序。

7.Repetitive Transformer Blocks 重复的 Transformer 模块

A Transformer block is the main unit of processing in the model. It has two parts:
Transformer 模块是模型处理的主要单元。它包含两个部分：
Multi-head self-attention-lets tokens share information
多头自注意力——让 token 共享信息
MLP-refines each token's details
MLP-细化每个 token 的细节
Models stack many blocks so token representations become richer as they pass through. GPT-2(small) has 12 of them.
模型堆叠多个模块，使得 token 表示随着通过它们而变得更加丰富。GPT-2(小)有 12 个这样的模块。

8. Multi-Head Self Attention 多头自我关注

Self-attention lets the model decide which parts of the input are most relevant to each token. This helps it capture meaning and relationships, even between far-apart words.
自注意力机制使模型能够判断输入中哪些部分与每个词元最相关。这有助于模型捕捉词义和关系，即使是相距甚远的词语之间也能建立联系。
In multi-head form, the model runs several attention processes in parallel, each focusing on different patterns in the text.
在多头模式下，该模型并行运行多个注意力过程，每个过程都专注于文本中的不同模式。

9. Query, Key, Value 查询、键、值

To perform self-attention, each token's embedding is transformed into three new embeddings—Query, Key, and Value. This transformation is done by applying different weights and biases to each token embedding. These parameters (weights and biases), are optimized through training.
为了执行自注意力机制，每个词元的嵌入被转换为三个新的嵌入—查询(Query)、键(Key)和值(Value)。这个转换是通过为每个词元嵌入应用不同的权重和偏差来完成的。这些参数(权重和偏差)通过训练进行优化。
Once created, queries compare with Keys to measure relevance, and this relevance is used to weight the Values.
创建后，查询(queries)与键(Keys)进行比较以衡量相关性，并使用这种相关性来对值(Values)进行加权。

10. Multi-head 多头

After creating Q, K, and V embeddings, the model splits them into several heads(12 in GPT-2 small). Each head works with its own smaller set of Q/K/V, focusing on different patterns in the text—like grammar, meaning, or long-range links.
在创建Q、K和V嵌入后，模型将它们分成多个头(GPT-2小模型中有12个)。每个头使用其自己的较小Q/K/V集，专注于文本中的不同模式—如语法、含义或长距离联系。
Multiple heads let the model learn many kinds of relationships in parallel, making its understanding richer.
多个头使模型能够并行学习多种关系，使其理解更加丰富。

11. Masked Self Attention 掩码自注意力

In each head, the model decides how much each token focuses on others:
在每个注意力头中，模型决定每个 token 对其他 token 的注意力程度：
Dot Product-Multiply matching numbers in Query/ Key vectors, sum to get attention scores.
点积-在Query/ Key 向量中匹配的数字相乘，求和得到注意力分数。
Mask-Hide future tokens so it can't peek ahead.
掩码-隐藏未来的 token，使其无法提前窥视。
Softmax-Convert scores to probabilities, each row summing to 1, showing focus on earlier tokens.
Softmax-将分数转换为概率，每行总和为 1，显示对早期 token 的关注。

12 .Attention Output & concatenation 注意力输出与拼接

Each head multiplies its attention scores with the Value embeddings to produce its attention output—a refined representation of each token after considering context.
每个注意力头将其注意力分数与值嵌入相乘，以产生其注意力输出——这是在考虑上下文后每个标记的精细化表示。
GPT-2(small) has 12 such outputs, which are concatenated to form a single vector of the original size (768 numbers).
GPT-2(小模型)有12个这样的输出，它们被连接起来形成原始大小的单个向量(768个数字)。

13. MLP (Multi-Layer Perceptron) MLP (多层感知器)

The attention output goes through an MLP to refine token representations. A Linear layer changes embedding values and size using learned weights and bias, then a non-linear activation decides how much each value passes.
注意力输出通过一个 MLP 来精细化标记表示。一个线性层使用学习到的权重和偏差改变嵌入值和大小，然后一个非线性激活函数决定每个值通过的程度。
Many activation types exist; GPT-2 uses GELU, which lets small values pass partially and large values pass fully, helping capture both subtle and strong patterns.
存在多种激活类型；GPT-2使用 GELU，它允许小值部分通过而大值完全通过，有助于捕捉微妙和强烈的模式。

14. Output Logit 输出逻辑值

After all Transformer blocks, the last token's output embedding, enriched with context from all previous tokens, is multiplied by learned weights in a final layer.
在所有 Transformer 模块之后，最后一个词的输出嵌入，融合了所有先前词的上下文信息，在最后一层被学习到的权重相乘。
This produces logits, 50,257 numbers—one for each token in GPT-2's vocabulary—that indicate how likely each token is to come next.
这产生了 logits，即 50,257 个数字——每个数字对应 GPT-2 词汇表中的每个词——表示每个词接下来出现的可能性。

15. Probabilities 概率

Logits are just raw scores. To make them easier to interpret, we convert them into probabilities between 0 and 1, where all add up to 1. This tells us the likelihood of each token being the next word.
Logits 只是原始分数。为了更易于理解，我们将它们转换为0到1之间的概率，所有概率之和为1。这告诉我们每个词作为下一个词的可能性。
Instead of always picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text.
我们不必总是选择概率最高的词，可以使用不同的选择策略来平衡生成文本中的安全性和创造性。

16. Temperature 温度

Temperature works by scaling the logits before turning them into probabilities. A low temperature (e.g., 0.2) makes large logits even larger and small ones smaller, favoring the highest-scoring tokens and leading to more predictable choices. A high temperature (e.g., 1.0 or above) flattens the differences, making less likely tokens more competitive and leading to more creative outputs.
温度通过缩放 logits 再将其转换为概率来工作。低温度(例如 0.2)会使大的 logits 更大，小的 logits 更小，从而倾向于得分最高的 token，导致更可预测的选择。高温度(例如 1.0或更高)会拉平差异，使不太可能的 token 更具竞争力，从而产生更具创造性的输出。

17. Sampling Strategy 采样策略

Finally, we need a strategy to pick the next token. Many exist, but here are common ones: Greedy search picks the top one. Top-k keeps only the k most likely tokens, and top-p keeps the smallest set whose total probability is at least p— trimming unlikely ones early.
最后，我们需要一个策略来选择下一个 token。存在许多策略，但这里介绍常见的几种：贪婪搜索选择得分最高的。Top-k保留最有可能的k个 token，而 top-p 保留总概率至少为 p 的最小集合——早期修剪不太可能的 token。
Then softmax turns the remaining logits into probabilities, and one token is picked at random from the allowed set.
然后 softmax 将剩余的 logits 转换为概率，并从允许的集合中随机选择一个 token。

18. Residual Connection 残差连接

Transformers have extra features that enhance the model performance but aren't core to understanding the basics. For example, a residual connection adds a layer's input to its output, keeping information and learning signals from fading through many layers. In GPT-2, it's used twice per block to train deeper stacks effectively.
Transformer具有增强模型性能但并非理解基础知识核心的额外功能。例如，残差连接将层的输入加到其输出上，防止信息和学习信号在多层中逐渐减弱。在GPT-2中，每个块使用两次来有效地训练更深层的堆栈。

19. Layer Normalization 层归一化

Layer Normalization helps stabilize both training and inference by adjusting input numbers so their mean and variance stay consistent. This makes the model less sensitive to its starting weights and helps it learn more effectively. In GPT-2, it's applied before self-attention, before the MLP, and once more before the final output.
Layer Normalization 通过调整输入数值，使它们的均值和方差保持一致，从而帮助稳定训练和推理过程。这使得模型对初始权重不那么敏感，有助于其更有效地学习。在GPT-2中，它应用于自注意力机制之前、MLP之前，以及最终输出之前。

20. Dropout

During training, dropout randomly turns off some connections between numbers so the model doesn't overfit to specific patterns. This helps it learn features that generalize better. GPT-2 uses it, but newer LLMs often skip it because they train on huge datasets and overfitting is less of a problem. In inference, dropout is turned off.
在训练期间，dropout会随机关闭一些数值之间的连接，以防止模型过度拟合特定模式。这有助于它学习更具泛化能力的特征。GPT-2使用dropout，但更新的LLM通常会跳过它，因为它们在大型数据集上训练，过度拟合问题不那么突出。在推理过程中，dropout会关闭。

标签