Lazy loaded image2.4 Continued Pre-training (CPT)

2.4 Continued Pre-training (CPT)

To obtain a large language model with outstanding capabilities in a specific vertical domain, it may be necessary to perform continued pre-training on data from that domain. Code and mathematics LLMs are two typical examples, such as Codex, Code LLaMA, DeepSeekMath, etc.

2.4.1 Fine-tuning or Continued Pre-training?

In papers such as Codex, Code Llama, and others, the process of further training a pre-trained model on code data is often fine-tuning rather than continued pre-training. These earlier works tended to adopt the paradigm from the BERT era, i.e., pre-training + fine-tuning, and thus chose to use the term fine-tuning. However, as time has passed, more and more work has opted for pre-training + continued pre-training + fine-tuning, especially when considering factors such as the volume of data used, the training objectives, and the optimization setup.
As mentioned in [3.1.1 Difference Between SFT and Pre-training], both the goal and training data format of pre-training are different from the SFT, where the former aims for knowledge injection. Here, adapting the model to another domain, like coding, is referred to as continued pre-training because we want the model first to absorb enough knowledge from that domain and then fine-tune it for instruction following.
📌
When is it necessary to continue pre-training?
It is necessary to continue pre-training when there is a significant difference in data distribution between the pre-training corpus and the downstream task corpus, such as general corpus v.s. code corpus. Note that the essence of such difference is token distribution difference because specific domains like legal, medical, coding, etc. usually have their own terminology.
For example: Suppose your current task is to perform Named Entity Recognition (NER) on legal documents, but the BERT model you're using is a general-purpose bert-base-cased. To achieve better performance on the downstream task, it's best to continue pre-training the model on a large amount of legal-domain text to obtain a pre-trained model that is better adapted to the legal field.

2.4.2 Continuing Pre-training at Vertical Domain

This usually refers to training a base model on domain-specific data to create a domain-specialized model, injecting domain knowledge — for example, models for code, legal texts, etc. The training process is the same as pre-training, but the data used and the amount of data are different. Generally, approximately general-domain data : domain-specific data : instruction data = 7:2:1.
  • General-domain data can come from parts of the base model's data.
  • Domain-specific data typically needs to be additionally collected, filtered, and selected. You can reuse the base model’s data preprocessing pipeline for cleaning and deduplication.
As we mentioned in [the Pretraining Tricks from 2.3.3 Pre-training Strategy], it’s common to add certrain ratio of instruction data during pre-training to boost its performance on later SFT stage. This is why in this CPT stage we add 10% of instruction data.
<ins/>

2.4.3 Long-text Continuing Pre-training

References: CodeLlama, Effective Long-Context Scaling of Foundation Models, YaRN
🌟
Example
Sample a 20B dataset for continued pretraining data, using the CodeLlama method for long-text continued pretraining and extrapolation method could be NTK-Aware.
  • Parameter Settings: Based on CodeLlama’s long context fine-tuning strategy, continuing from a previously trained language model base, long-text data is used for continued pretraining
    • Continue pretraining with a sequence length extended from 4096 to 16384
    • Adjust the RoPE parameter so that the increases from 10,000 to 1,000,000. This reduces the attenuation effect of RoPE on distant tokens, thereby helping mitigate the decay of attention scores and allowing the model to focus more on long-range dependencies.
  • Pretraining Acceleration
      • For long-text input, apply context parallel parallel training. Sequence input is segmented according to model input length, similar to the principle of ring attention/flash attention, significantly improving model throughput (by 50%+ on 32k/128k training), speeding up long-text training.
Prev
2.3 Pre-training Workflow
Next
2.5 Pre-training Evaluation
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant