2.2 Pre-training Data

2.2 Pre-training Data

2.2.1 Data Source

• Common Crawl: An open data-sharing platform containing massive amounts of web page data.

• GitHub: Code-related data collected from GitHub repositories.

• Web text: Text from code-related and math-related websites, such as programming forums, library documentation, and math resource sites.

• E-books, Educational Materials, and Research Papers: These resources contain rich knowledge, enhancing the model's understanding and generation capabilities across various domains.

• Internal Data: Proprietary data curated by companies developing large models, often including vertical business-critical information.

• Open-Source General Datasets

• Example of Continued Pre-training:
• General data: 70B tokens
• Total continued pre-training data: 100B tokens

• Chinese Corpus

• English Corpus

• Code Corpus

• English/Chinese/Code Percentage
• 7:2:1
• In reference to Yi’s

2.2.2 Data Crawling

Example of Targeted Website Crawling

Example of Keywords-based Crawling

• Basic configuration

• Run script

2.2.3 Data Cleaning

URL Filtering (llama 3.1)

Content Extraction

• Note: you may need to install the libs first

Language Identification

1. Use FastText to train a language identification model, and discard articles with a language confidence score below 0.65.

2. Use langid to identify the language type.

3. language identification using data-juicer. Below is the key code snippet.

Low-Quality Filtering

• Article-Level Filtering: First, remove some articles that repeatedly contain the same content or contain incorrect information (e.g., scraped content, etc.); secondly, for each article, determine the overall length, the proportion of punctuation in the article, etc., to filter out irregular articles.

• Sentence-Level Filtering: Filter out some useless sentences in the article. These types of sentences usually have no specific meaning, are conversational, and hard to classify. Therefore, use the following heuristics to filter such sentences (some rules are only applicable to English):
🧸
1. Sentence is mainly composed of uppercase letters (discard)
2. Sentence is composed of numbers/digits (discard)
3. Contains pre-defined keywords, such as “click here,” “forward,” “like” (discard)
4. If the sentence length is less than 10, and it starts or ends with the following (discard):
• Starts with “login, register”
• Ends with “more, expand”

• Use heuristic methods to perform pre-train data quality filtering, such as determining whether data length is below a certain threshold; whether the proportion of a certain token in the data exceeds a certain threshold; the proportion of Chinese, English, digits; whether it contains "http"; whether it contains low-quality keywords like COVID, "epidemic", etc.; whether it contains strong sentiment or anti-society content; whether it includes explicit words. Note that while applying these rules, do not overdo such that the data distribution is biased.

You can see that data processing is a very meticulous job: some things can be automated and fast through training models, but model outputs can easily introduce bias, so some engineers still use rule-based filtering instead of models to clean the data.

Model Scoring

Data De-duplication

• Duplication within training datasets: Refers to duplication within and between different sources of training data.
• Intra-dataset duplication: including duplication within a single document across lines, paragraphs, or n-grams (suggesting low data quality); and inter-dataset duplication, referring to overlaps between multiple documents (whether exact or fuzzy matches).
• Inter-dataset duplication: often caused when multiple datasets are derived from the same source but preprocessed differently. A typical example is LLaMA, which uses both preprocessed CommonCrawl data and previously preprocessed C4 data from CommonCrawl used in T5, where both are from CommonCrawl and may have significant overlap.

• Duplication in training configuration: Refers to redundancy caused by how many epochs (repetitions) are used to train different types, sources, or qualities of data. Usually, the number of training epochs varies across datasets.

• Duplication between training and test sets: For instance, datasets used for monitoring Language Modeling Loss during pre-training or for benchmarking should be deduplicated against the training set to avoid overlap.

Exploring Data Diversity

Practical Vertical Domain Experience

1. Acquire as many data documents as possible to serve as the initial seed data corpus.

2. Use clustering methods to filter domain-related data from the Chinese text corpus. Retrieve relevant domain documents from the internet (encyclopedias, web data, books, etc.) to expand the seed data.
Pipeline: Keyword filtering of general data → Retrieve top 10 most similar domain documents → Human filtering
1. Keyword filtering of general data
1. First, preprocess the initial seed data corpus to preserve paragraph structure, segment content by heading, and split knowledge document data.
2. Use Jieba to extract keywords.
3. Filter mined keywords manually and compile a final keyword list.
4. From the retrieved general data from the internet, filter documents that contain N keywords.
2. Retrieve top 10 data most similar to the initial seed data corpus
1. Use the first 512 tokens of the general document to compute similarity.
3. Human filtering
1. Manually review the general data to select domain-related documents to expand the initial seed data.

3. Train a Chinese domain data classifier
1. Training data: Use vertical domain initial seed data and expanded seed documents from step 2 (each with first 1k tokens) as positive samples. Negative samples are "non-vertical-domain" documents.
2. Model: A BERT model with 1024 hidden dimensions.
3. Model performance: Acquire a high precision (95%) trained classifier to classify domain-relevant text from the Chinese corpus, extracting 0.5 B of vertical data (B = billion).

2.2.4 Data Ratio and Order

• Guiding Principle: Classify according to task type — the finer the classification, the better. For example: Chinese, English, and code. Each document should be classified (e.g., into Wikipedia, code, markdown, etc.). You can still choose to use BERT-like models for classification.

• In many open-source LLM tech reports, the training data is often divided into Knowledge + Code + Reasoning. Among them, knowledge data is mostly Chinese and English, reasoning data includes math and chain-of-thought (CoT) mixed data. In general, Chinese model data proportions are often about: Chinese : English : Code = 4 : 4 : 2. Chinese LLMs should not rely too heavily on Chinese (half or more), and English should not be too little either. If you want strong reasoning capabilities, you must add as much math and code data as possible.

• Best Practice: If you're doing domain-specific model pretraining, consider a more refined domain data ratio for better continue pre-training. If you want the model to retain general capabilities (like summarization or QA), then:
• Domain data should not exceed 15% of total data. If you exceed this, model generalization may drop. This threshold depends on the base model's size, e.g., llama may need a smaller ratio. Empirically — recommended range: 10%-15%. Practice is required for different vertical domains, LLM models, etc.
• In SFT, then Domain : General = 1 : 1 is also okay. If SFT data is too few, you may not have to mix them with general data at all.

• The above data ratios are usually verified with small-scale pretraining experiments, then scaled up using scaling laws, so small models are typically used for ablation and performance validation.

• Most large-scale models usually rely on classes, i.e. what to learn first and what to learn next. Pay attention to the training data order during this stage to prevent performance degradation caused by catastrophic forgetting. This part should also be experimented on small models first.