4.3 GPT Series

4.3 GPT Series

4.3.1 GPT1

• GPT (Generative Pre-Training) Architecture: Decoder-only Transformer with 12 layers. The architecture is similar to the standard transformer decoder but with trainable positional embeddings. In the original Transformer, the decoder includes two types of attention: Cross-attention, where keys K and values V come from the encoder, and queries Q come from the decoder, and Masked multi-head attention. But GPT uses only masked multi-head attention.

• Training Paradigm: Self-supervised Pre-training + Supervised Fine-tuning. The key idea is unsupervised learning.
• Pre-training Objective Function: Predicting the next word based on the previous K tokens:
• Fine-tuning Objective Function: Full input sequence combined with task-specific labels. The supervised objective is added on top of the unsupervised one. Let y be the label, the loss is defined as:
The combined loss becomes:
Purpose of the unsupervised loss: 1) Improve generalization of the SFT (Supervised Fine-Tuning). 2) Speed up convergence
• Modified Input Format: Special tokens like [Start] and [Extract] are added to mark the beginning and end of the sequence. [Delim] is inserted between sequences to indicate separation. The last token in the last layer is passed to the task-specific head, enabling the model to handle various downstream tasks.

4.3.2 GPT2

• Model Architecture: Largely the same as GPT-1, but with post-norm replaced by pre-norm, input sequence length increased from 512 to 1024, and has 48 layers for the largest GPT2 model.

• Training Paradigm: Pre-training + zero-shot prompting.
• The learning goal is to use an unsupervised pre-trained language model to perform supervised tasks. The underlying idea is like, once a language model is large enough, it can cover all supervised tasks within its unsupervised pre-training. In other words, supervised learning becomes a subset of unsupervised language modeling. For example, once the model has seen the sentence "Michael Jordan is the best basketball player in the history" during pre-training, it can also handle the Q&A task: Question: "Who is the best basketball player in the history?" Answer: "Michael Jordan".
• GPT-2 can perform downstream tasks in a zero-shot setting, meaning no labeled data or fine-tuning is required.
• To enable zero-shot, the input format for downstream tasks must not include special tokens for start, separation, or ending like GPT-1 does. Instead, the input should look like the natural language the model has seen during pre-training.
• Zero-shot can be done through prompting. For example, for machine translation or reading comprehension, a prompt might be: "Translate the following sentence into French, English, or Chinese."
• Why does zero-shot prompting work? Because the pre-training dataset is so large and diverse, it likely includes many examples of prompts and instructions related to various tasks across different domains. As a result, the model naturally has some zero-shot capabilities during training.

• Key Idea: Any supervised task is a subset of language modeling. When a model is sufficiently large and trained on a rich enough dataset, it can complete other supervised learning tasks just by unsupervised language modeling.

• Training Data: High-quality documents scraped from Reddit, totaling 8 million documents and a total of 40 GB.

• Differences from GPT-1:
• Model Architecture: LayerNorm position was adjusted; parameter initialization was also modified.
• GPT-2 emphasizes zero-shot, whereas GPT-1 focuses on pre-train + fine-tune.
• Data size increased: GPT-2 used 40 GB, and GPT-1 used 5 GB.
• Model scale increased: The largest GPT-2 model has 1.5 billion parameters, compared to 110 million for GPT-1.

4.3.3 GPT3

• Model Architecture: Similar to GPT-2, but introduces Sparse Attention.
• Dense Attention: Each token computes attention with every other token. Complexity:
• Sparse Attention: Each token computes attention with only a subset of other tokens. Complexity:
Advantages of Sparse Attention:
• Reduces the computational complexity of attention layers, saves memory and runtime, enabling processing of longer input sequences
• Captures both local dense and long-range sparse dependencies: More attention to nearby context, less to distant context

• Training Paradigm: Pre-training + few-shot / in-context learning

• Differences to GPT-2:
• Architecture: Built upon GPT-2, but replaces the attention mechanism with sparse attention.
• Performance: Significantly outperforms GPT-2 in terms of generation quality and realism.
• Learning Paradigm: GPT-3 emphasizes few-shot, while GPT-2 focuses on zero-shot.
• GPT-3 uses much more training data:
• GPT-3: 45 TB (raw), 570 GB (cleaned)
• GPT-2: 40 GB
• Model Size:
• GPT-3’s largest model has 175 billion parameters
• GPT-2’s largest model has 1.5 billion parameters