The evolution of ChatGPT in recent years is remarkable.

“An Unchanging Core, Evolving Words—The Prototype of ChatGPT and GPT-2”

The evolution of ChatGPT in recent years is remarkable. Its ability to handle natural conversation, complex reasoning, and even code generation creates the illusion that it is approaching human intelligence. However, it is surprisingly little known that the fundamental computational principle at its core has hardly changed since the days of GPT-2.

GPT-2 is a language model based on the “autoregressive transformer” that appeared in 2019. The combination of this simple mechanism, which predicts tokens one by one, and the attention mechanism known as multi-head attention, was already extremely powerful at that time. Current ChatGPT models, such as GPT-3.5 and GPT-4, are fundamentally nothing more than “scaled up” and “refined” versions of this GPT-2 structure.

Certainly, many improvements have been made to models since GPT-2. For example, improvements to positional embeddings to handle long texts, changes to normalization methods to improve training stability, and innovations in attention to increase computational efficiency. However, these are merely adjustments piled on top of the foundational “transformer” structure, and the principle itself has not been overturned.

In other words, current ChatGPT is a massive tower built as an extension of the philosophy of GPT-2. The foundation remains unchanged; it is just that the stones piled upon it have increased, been polished, and the tip of the tower has come to pierce the clouds.

That is why we should marvel at the evolution of AI while also turning our eyes to its “immutable structural beauty.” Innovation is not always about creating something new, but also an attempt to delve deeper into what already exists and make it better.

And that is the essence of the history of AI, starting from GPT-2 and leading to ChatGPT.

📝 Notes (Not a prerequisite for execution)

• This code is a proof of concept for “what happens when you scale the GPT-2 structure”.

The objective is to "simulate the construction of a scalable autoregressive language model with 8B to 70B parameters while maintaining the GPT-2 structure."
Execution is unrealistic, and this is template code for conceptual design.

🧠 GPT-2 scaling language model construction code (with Japanese comments)

Code (with Japanese comments) treasure code.

import torch
import torch.nn as nn
import math
from dataclasses import dataclass
from typing import Optional

# =========================
# 設定クラス：モデルの各種ハイパーパラメータを定義
# =========================
@dataclass
class GPT2Config:
    vocab_size: int = 50257  # 使用する語彙数（トークナイザーに依存）
    max_position_embeddings: int = 2048  # 最大トークン長（文の長さ）
    n_layer: int = 48          # Transformer層の数（例：8B用に48層）
    n_embd: int = 3072         # 隠れ状態の次元数（embeddingのサイズ）
    n_head: int = 48           # Attentionのヘッド数（通常 n_embd // 64）
    dropout: float = 0.1       # ドロップアウト率（過学習防止）

# =========================
# GPT-2型の自己注意機構（Multi-Head Attention）
# =========================
class GPT2Attention(nn.Module):
    def __init__(self, config: GPT2Config):
        super().__init__()
        self.embed_dim = config.n_embd
        self.num_heads = config.n_head
        self.head_dim = self.embed_dim // self.num_heads  # 1ヘッドあたりの次元数
        self.scale = self.head_dim ** -0.5  # Attentionスコアのスケーリング

        # Query, Key, Value をまとめて線形変換
        self.qkv_proj = nn.Linear(config.n_embd, 3 * config.n_embd)
        self.out_proj = nn.Linear(config.n_embd, config.n_embd)

    def forward(self, x):
        B, T, C = x.size()  # バッチサイズ, シーケンス長, 埋め込み次元
        qkv = self.qkv_proj(x).chunk(3, dim=-1)  # Q, K, Vに分割
        q, k, v = (t.view(B, T, self.num_heads, -1).transpose(1, 2) for t in qkv)

        # Attentionスコア計算： (Q × K^T) / √d
        attn_scores = (q @ k.transpose(-2, -1)) * self.scale
        attn_weights = attn_scores.softmax(dim=-1)

        # Attention出力 = 重み付き平均(V)
        attn_output = attn_weights @ v

        # 結果を結合して元の次元に戻す
        attn_output = attn_output.transpose(1, 2).contiguous().view(B, T, C)
        return self.out_proj(attn_output)

# =========================
# Feed Forward層（MLP）：中間層→活性化→出力層
# =========================
class GPT2MLP(nn.Module):
    def __init__(self, config: GPT2Config):
        super().__init__()
        # GPT-2では中間層のサイズは通常 4 × n_embd
        self.fc_in = nn.Linear(config.n_embd, 4 * config.n_embd)
        self.fc_out = nn.Linear(4 * config.n_embd, config.n_embd)
        self.act = nn.GELU()  # 非線形活性化関数（滑らかなReLU）

    def forward(self, x):
        return self.fc_out(self.act(self.fc_in(x)))

# =========================
# Transformerブロック（Attention + MLP + 残差 + LayerNorm）
# =========================
class GPT2Block(nn.Module):
    def __init__(self, config: GPT2Config):
        super().__init__()
        self.ln1 = nn.LayerNorm(config.n_embd)
        self.attn = GPT2Attention(config)
        self.ln2 = nn.LayerNorm(config.n_embd)
        self.mlp = GPT2MLP(config)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))  # 残差接続 + 注意
        x = x + self.mlp(self.ln2(x))   # 残差接続 + FFN
        return x

# =========================
# GPT-2本体：Embedding + Position + N層Block + 出力層
# =========================
class GPT2Model(nn.Module):
    def __init__(self, config: GPT2Config):
        super().__init__()
        # トークン埋め込み（入力語彙をベクトルに変換）
        self.token_embedding = nn.Embedding(config.vocab_size, config.n_embd)

        # 位置埋め込み（各トークンの位置をエンコード）
        self.position_embedding = nn.Parameter(torch.zeros(1, config.max_position_embeddings, config.n_embd))

        # Transformerブロックを複数層作成
        self.blocks = nn.ModuleList([GPT2Block(config) for _ in range(config.n_layer)])

        # 最終LayerNormと出力（語彙予測）線形層
        self.ln_f = nn.LayerNorm(config.n_embd)
        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

    def forward(self, input_ids):
        B, T = input_ids.size()
        # 入力トークンに埋め込み＋位置情報を加算
        x = self.token_embedding(input_ids) + self.position_embedding[:, :T, :]

        # 各Transformerブロックを順に適用
        for block in self.blocks:
            x = block(x)

        # 最終出力層で語彙ごとのスコアを出力（ロジット）
        x = self.ln_f(x)
        return self.lm_head(x)

# =========================
# トレーニング関数（簡易版）
# =========================
def train(model, optimizer, dataloader):
    model.train()
    for batch in dataloader:
        input_ids = batch['input_ids'].to(model.device)
        targets = batch['labels'].to(model.device)

        # モデルの順伝播（ロジット出力）
        logits = model(input_ids)

        # 損失関数（トークンごとのクロスエントロピー）
        loss = nn.CrossEntropyLoss()(logits.view(-1, logits.size(-1)), targets.view(-1))

        # 誤差逆伝播とパラメータ更新
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

# =========================
# 評価関数（検証用）
# =========================
def evaluate(model, dataloader):
    model.eval()
    total_loss = 0
    with torch.no_grad():
        for batch in dataloader:
            input_ids = batch['input_ids'].to(model.device)
            targets = batch['labels'].to(model.device)
            logits = model(input_ids)
            loss = nn.CrossEntropyLoss()(logits.view(-1, logits.size(-1)), targets.view(-1))
            total_loss += loss.item()
    return total_loss / len(dataloader)