4.6 Qwen Series

4.6.1 Qwen1

Paper: QWEN TECHNICAL REPORT

Model Architecture: a modified Transformer architecture, similar to LLaMA.

Uses tiktoken BPE, with the cl100k_base vocabulary as the starting point to extend the vocabulary. It also splits numbers into individual digits. Final vocabulary size is approximately 152K.
Untied Embedding: Input embedding and output embedding are not shared—two separate embedding matrices are used. This increases memory consumption but significantly improves model performance.
Uses RoPE positional encoding, Pre-RMSNorm, and SwiGLU activation function.
Bias term is removed in most layers, but retained in the QKV layers. This helps improve the model's extrapolation capability. Adding bias terms allows the model to better capture input features, thereby enhancing its performance on unseen or out-of-distribution data. This strategy to strengthen extrapolation is crucial for handling complex tasks and diverse data inputs.

Model Training:

Uses the standard autoregressive language modeling objective.
Context length during training is 2048 tokens. To build training batches, document text is randomly shuffled and concatenated, then truncated to the target length.
The attention module uses Flash Attention to improve computational efficiency and reduce memory usage.
Training is accelerated using BF16 mixed precision.
The optimizer is AdamW, with hyperparameters: , ,
Implements a cosine learning rate decay schedule, with the learning rate decaying to 10% of the peak value at the end of training.

Context Length (Extrapolation) Extension: Techniques such as NTK-aware interpolation, LogN-Scaling, and Layer-wise Windowed Self-Attention are used to effectively extend the model's context length while preserving performance.

NTK-aware interpolation: check Section 1.5.5: Length Extrapolation.
LogN-Scaling: LogN-Scaling rescales the dot product of the query q and value v by a factor that depends on the ratio of the context length to the training length, ensuring that the entropy of the attention value remains stable as the context length grows.

Layer-wise Sliding Window Attention: Window attention restricts the attention to a limited context window, preventing the model from attending to tokens that are too far away. Different window sizes are used across layers—lower layers use smaller windows, and higher layers use larger ones.

Combining the above-mentioned techniques, Qwen was pre-trained with a sequence length of 2048, and during inference it can handle sequences of up to 8192.

Reward Model Training:

Preference model pretraining: The reward model is initialized using a Qwen model of the same size combined with pooling layers. Special sentence-ending tokens are mapped via the pooling layer as reward scores.
Reward model fine-tuning:

Classification System: To ensure diversity and complexity in prompts, a classification system with around 6,600 detailed labels was constructed.
Balanced Sampling: A balanced sampling strategy was adopted to select prompts in a way that accounts for both diversity and complexity.
Diversity Sampling: To generate diverse responses, the experiment utilized Qwen models of varying scales and sampling strategies, as diversified responses help reduce labeling difficulty and improve the performance of the reward model.

<ins/>

4.6.2 Qwen2

Qwen2 Technical Report

Link: https://arxiv.org/abs/2407.10671

Model Architecture:

Differences from Qwen include GQA (Grouped Query Attention), YaRN (Yet another RoPE extensioN) + Dual Chunk Attention (DCA)

For YaRN, check Section 1.5.5, “Length Extrapolation Optimization,”

For DCA, check Section 1.3.6, “DCA (Dual Chunk Attention),”

Tokenizer: BBPE, vocabulary size of 151,643

Model Training:

Pre-training

Quality Enhancement: The filtering algorithm has been refined with additional heuristic and model-based methods, including the use of the Qwen models to filter out low-quality data. Moreover, these models are utilized to synthesize high-quality pre-training data.
Data Expansion: Compared to Qwen1.5, It collected a significantly larger amount of high-quality code, mathematics, and multilingual data, enhancing the model’s capabilities in respective areas. This new dataset supports approximately 30 languages, such as English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, and Vietnamese.
Distribution Improvement: To ensure the model learns the distribution similar to human, experiments were conducted on scaled-down models to optimize the mixing of data from various sources and domains.

Data Synthesis for Post-training

Rejection Sampling: For mathematical or similar tasks with definitive final answers, rejection sampling is applied to improve the quality of solutions. Large language models (LLMs) are tasked to generate multiple responses for each instruction, namely the reasoning paths. Paths that result in accurate conclusions and are considered reasonable by the model are preserved, serving as demonstration data. Preference data is generated by comparing correct and incorrect paths.
Execution Feedback: For coding tasks, LLMs are employed to generate solutions and associated test cases. The efficacy of these solutions is evaluated by compiling and executing them against the test cases, thereby creating demonstration and preference data. This methodology is also applicable to assessing instruction following. For each instruction with constraints, e.g., length limit, the LLM is tasked to generate a Python verification function to ensure the response aligns with the instruction requirements.
Data Repurposing: Creating high-quality responses in literary writing tasks is challenging for annotators without specialized training. To tackle this problem, we aggregate high-quality literary works from the public domain and employ LLMs to develop instructions with varying levels of detail. These instructions, paired with the original works, serve as demonstration data.

For example, to compile roleplay data with vivid and engaging responses, we source detailed character profiles from knowledge repositories such as Wikipedia and instruct LLMs to generate corresponding instructions and responses. This process, similar to a reading comprehension task, ensures that the integrity of the character’s profile is maintained.

Constitutional Feedback: Constitutional AI refers to the process of guiding LLMs to generate responses based on predefined sets of principles. To ensure adherence to guidelines such as safety and values, a constitution dataset was compiled. This dataset delineates principles to be followed and those to be avoided. It was used to instruct LLMs to produce responses that either are aligned with or deviated from these guidelines, serving as a reference for demonstration and preference data.

RLHF Training Process: offline and online training.

Offline Training Stage: A pre-compiled preference dataset P was used to maximize the difference in likelihood between and with Direct Preference Optimization (DPO)
Online Training Stage: the model iteratively refines its performance in real-time, leveraging reward models for immediate feedback. Specifically, multiple responses was sampled from the current policy model, and the reward model selects the most and the least preferred responses, forming preference pairs that are used for DPO in each episode. Moreover, Online Merging Optimizer was employed to mitigate the alignment tax, i.e., the performance degradation associated with aligning model generation with human preferences.