4.4 LLaMA Series

4.4 LLaMA Series

4.4.1 LLaMA1

• Model Architecture: Compared to GPT, LLaMA introduces the following modifications:
• To improve training stability, LLaMA adopts pre-RMSNorm as the layer normalization.
• To enhance model performance, it uses SwiGLU (Swish-Gated Linear Unit) as the activation function, where Swish is a.k.a. SiLU.
• To better capture long-range dependencies, it uses RoPE (Rotary Positional Embedding) for positional encoding.
• To more effectively tokenize data, LLaMA-1 uses the Byte-level BPE algorithm for tokenization, which is implemented using SentencePiece. A noteworthy detail is that LLaMA-1 treats all digits as individual tokens. The vocabulary size is 32K.

• Training Data: LLaMA is trained using a large amount of unlabeled data in a self-supervised manner, totaling 1.4T tokens. The pretraining data is a mixture of multiple sources, all of which are publicly available datasets. The amount and sampling ratio of data from each source are shown in the table below.

• Training Strategy: LLaMA is trained using a basic self-supervised learning setup, without any form of task-specific fine-tuning.

• LLaMA-1’s technical report provides detailed training configuration using the AdamW optimizer, a variant of the Adam optimizer that improves training stability by handling weight decay more effectively. The choice of parameters β1 and β2 impacts the convergence behavior and stability during training.

• The cosine learning rate schedule described in LLaMA-1 is an effective technique for adjusting the learning rate during training. By gradually decreasing the learning rate, it can lead to better convergence in some cases. Using a weight decay of 0.1 and clipping gradients of 1.0 are standard practices to prevent overfitting and ensure numerical stability.

• Warmup is a heuristic strategy used to stabilize training in the early phase. It gradually increases the learning rate and/or batch size, serving as a practical approach to optimize resource allocation and efficiency, with potential benefits for model performance.

4.4.2 LLaMA2

• Model Architecture:
• Compared to LLaMA1, LLaMA2 adopts GQA (Grouped Query Attention) in large parameter models, which reduces the overall parameters.
• The FFN (Feedforward Network) projection dimension is expanded, improving model expressiveness and increasing the total number of parameters.
• Other improvements include more training data and a longer context window of up to 4K tokens (LLaMA1 was 2K).

• Training Data: LLaMA-2 is pre-trained on 2T tokens of publicly available data sources (specific datasets are not disclosed). Poor-quality data has a negative impact — it’s better to have less high-quality data than more low-quality data (Quality over quantity), highlighting the importance of careful data curation.

• Training Procedure: By incorporating human feedback and rejection sampling, the model can more effectively utilize scarce high-quality data, thereby maximizing training efficiency and effectiveness.

⭐

Reject Sampling is a fundamental Monte Carlo method used to sample from a target probability distribution. It is typically applied when it is difficult to sample directly from the target distribution. Reject sampling works by drawing samples from an easier proposal distribution—such as a uniform or normal distribution—and accepting them based on a criterion that ensures the accepted samples follow the target distribution.

⭐

In the context of LLMs: Generating K responses from the model for the same prompt, and using a Reward Model to score these responses, selecting the highest-scoring and best-performing one. This process not only improves the quality of generated responses, but also helps produce higher-quality samples for further training stages.

• Post-Training Summary:
• SFT Stage: The Supervised Fine-Tuning (SFT) phase should not last too long. Annotators may not be highly skilled, and the quality of their outputs is often not optimal. Typically, around 10,000 samples are enough for the model to reach the annotator’s level.
• Constructing Reward Model’s Training Data:
• It’s better if the data comes from its own model outputs (regardless of whether it’s 7B, 13B, or 34B; they likely share similar pretraining data distribution). This makes sense because the goal of a reward model is to help improve your own model. If your training data all comes from outputs of, say, GPT-4, then your reward model can only learn to judge GPT-4 outputs. But those outputs are distributionally different from your own model’s, which reduces effectiveness.
• Use different sampling strategies (e.g., temperature settings) for models of different sizes, then perform manual labeling on the sampled responses.
• Training is gradual, not once and for all. Since the training data for reward models is generated by the current version of the model, the reward model probably can only guide the current model to achieve small and incremental improvements. That’s why continuous labeling is needed. In LLaMA 2, this process was repeated 5 times, with substantial improvement in model performance after each iteration.



• When PPO (Proximal Policy Optimization) or other reinforcement learning algorithms are used is important: In LLaMA 2, the first four iterations used a simple BoN (Best-of-N) strategy, with only the final iteration employing PPO. This suggests the same thing as DeepSeek, which applies SFT first, then GRPO.

4.4.3 CodeLlama

• Training pipeline:

• The objective of the infilling task for CodeLLaMA 7B and 13B models: Predict the missing code segments based on the surrounding code context.
• Select a portion of code from a complete snippet and apply masking, replacing it with the <MASK> token to form the surrounding context.
• Use an autoregressive approach to predict the masked code segment based on the context.

4.4.4 Llama3

• The paper includes two model series: Llama3 and Llama3.1. The differences are minor—Llama3.1 shows improvements in multilingual capability, long-context handling, and tool use. The main experimental results in the paper are based on Llama3.1.

• Model architecture
• Compared to LLaMA2, LLaMA3 replaces the tokenizer from SentencePiece to TikToken, consistent with GPT-4. The vocabulary size is expanded from 32k to 128k.
• Both LLaMA3-8B and LLaMA3-70B adopt GQA (Grouped Query Attention).
• The context length is extended to 8k initially (and up to 128k in later pre-training stages using multi-stage long text).

• Training Data
• The pretraining corpus was carefully curated and significantly expanded in both size and quality, reaching 15T tokens, an ~8× increase. Among this, code data was expanded 4×, resulting in substantial improvements in code and reasoning performance.
• Notably, LLaMA-3 does not use the MOE (Mixture of Experts) architecture, which is typically used to reduce training/inference cost, though it often lags behind dense models in performance at the same scale. As model size continues to grow, how to reduce inference cost will become a key challenge.
• In addition, LLaMA-3’s training data includes a large amount of code tokens and over 5% non-English tokens, covering more than 30 languages. This improves the model's efficiency in handling English and enhances multilingual processing capabilities.
• To ensure data quality, Meta developed a series of data filtering pipelines, including heuristic filtering, NSFW filtering, semantic deduplication, and a quality classifier to assess the quality of candidate data. These tools benefit from performance improvement of previous LLaMA versions, especially in terms of indentifying high-quality data.
• Finally, Meta conducted large-scale experiments to evaluate the performance of mix data sources for pretraining, enabling LLaMA-3 to demonstrate strong performance across various domains and tasks, such as daily matters, STEM fields, programming, and historical knowledge.

• Training Pipeline: The LLaMA-3 series includes two model types — the pre-trained model (Llama-3) and the instruction-tuned model (Llama-3-Instruct).
• Pretraining Phase: To effectively utilize pretraining data, LLaMA-3 invested heavily in large-scale pretraining. Specifically, a series of scaling laws were created for downstream benchmarking, allowing the prediction of model performance on key tasks before training began and the selection of the optimal data mixture accordingly.
Overall process: 1) Initial pretraining, 2) Long-context pretraining, 3) Annealing
• Post-training Phase: A combination of Supervised Fine-tuning (SFT), rejection sampling, Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO) was used for multi-round alignment. The quality of SFT prompts and preference ranking used in PPO/DPO play major roles in performance. In fact, some of the biggest improvements in LLaMA-3's model quality came from carefully selecting the data and running multiple rounds of human-led quality reviews.

1. Reward Model: On top of the pre-trained model, a reward model is trained using human-annotated preference data. The goal is to assess whether the generated text aligns with human expectations.

2. SFT (Supervised Fine-Tuning): The reward model is used to filter the best outputs from collected prompts (rejection sampling). These filtered samples, along with other data sources (including synthetic data), are used to fine-tune the base model with a cross-entropy loss.

3. DPO (Direct Preference Optimization): Building upon the SFT-ed model, further training is conducted using the latest human preference data, with a focus on improving the model’s ability to follow instructions accurately.

4. Model Averaging: This involves averaging all the versions of models trained with different versions of data or hyperparameters, ending up with an averaged model version of the reward model (RM), the SFT-ed model, and the DPO-tuned model. The goal is to produce more stable and reliable outputs than relying on a single model. Model averaging also helps reduce overfitting to specific training samples, improving the model’s generalization ability on unseen data.

• LLaMA-3’s Improvements to DPO:
• Masking out formatting tokens in DPO loss: It masks out special formatting tokens including header and termination tokens from both chosen and rejected responses in the loss to stabilize DPO training, because authors observe that having these tokens contribute to the loss may lead to undesired model behaviors such as tail repetition or abruptly generating termination tokens. Authors hypothesize that this is due to the contrastive nature of the DPO loss – the presence of common tokens in both chosen and rejected responses leads to a conflicting learning objective as the model needs to increase and reduce the likelihood of these tokens simultaneously.
• Regularization with NLL loss: The author add an additional negative log-likelihood (NLL) loss term with a scaling coefficient of 0.2 on the chosen sequences. This helps further stabilize DPO training by maintaining the desired formatting for generation and preventing the decrease of the log probability of chosen responses.