3.1 Supervised Fine-Tuning (SFT)

3.1 Supervised Fine-Tuning (SFT)

3.1.1 Difference Between SFT and Pre-training

• Pretraining: Each data sample is of full length, reaching the model's maximum input length limit. SFT, however, retains the original length of each data sample without packing (i.e., no concatenation of samples).

• SFT introduces special tokens that have never been used during pretraining to teach the model new semantics. These tokens can be used to "construct knowledge," such as <imd_start>like<imd_end>. Special tokens help segment the corpus into different roles, typically including system, user, and assistant. Additional roles like tool, background, narration, or event may be added based on business needs.

• SFT introduces the most important token eos_token (end-of-sequence token), which pre-trained models have not encountered and thus cannot halt token generation.

• In SFT, the loss is not computed for the prompt portion of the output. This is because prompts are often highly homogeneous. Without loss masking for them, the same prompt would be redundantly learned. However, if every prompt is guaranteed to be unique, the loss masking for prompts can be omitted. For multi-turn dialogue data, it is essential to decide whether to compute the loss for every answer or only for the final answer.

• Pretraining focuses on memorization—purely absorbing broad knowledge from diverse data—while Supervised Fine-Tuning (SFT) emphasizes problem-solving, teaching the model to follow instructions and generate task-specific outputs. Critically, avoid injecting domain knowledge (e.g., 500K lines of code) directly during SFT, as this risks catastrophic overfitting. Instead, knowledge injection should occur via continue-pretraining (the model before doing SFT), where domain-specific data is blended into the pretraining corpus at a controlled ratio (e.g., 10–20%). This preserves the model’s general capabilities while adapting it to new specific domains.

1. Look for a Base (Raw) Checkpoint: If it exists or is partially released, see if you can get the “base” model from the same family. For example: DeepSeek-V3-Base.

2. Continue-Pretrain from the Instruction-Tuned Checkpoint: If the “raw” base model is truly not available, you can still attempt “continued pretraining” from the final SFT checkpoint. This is not as ideal as we mentioned before. To mitigate these risks, we may 1) use a moderate ratio of domain text vs. general text. 2) monitor perplexity on a validation set that includes both domain data and some general data.

3. Consider partial or Parameter-Efficient Finetuning (PEFT) strategies. Please check section 3.7 for detailed information.

3.1.2 SFT Data and Processing

Industry Consensus

1. Prompt quality and diversity are far more important than dataset size. Fine-tuning a 30B-parameter base model typically requires only around 100K high-quality examples.

1. Synthetic data is crucial! It should be generated through multiple diverse methods to reduce bias.

1. Including pretraining data can mitigate catastrophic forgetting (or catastrophic interference).

1. Train for one epoch in most cases. For niche domains with limited data, train for up to 3 epochs to overfit the data.

2. Prefer full fine-tuning over PEFT (Parameter-Efficient Fine-Tuning) whenever possible.

3. Avoid excessive knowledge injection during SFT. Overloading the model with domain-specific data beyond its capacity can lead to alignment tax and a decline in general performance.

Data Flywheel

1. Collect user prompts from a recent time period (e.g., two weeks)

2. Filter them with heuristic rules for quality, privacy, etc.

3. Label or generate high‐quality answers using a more powerful model (e.g., GPT‐4o)

4. Collect the resulting (prompt, answer) pairs

1. Data Collection. Aggregate data from multiple sources, including user interactions, social media analytics, and direct signals like upvotes/downvotes. Continuously pull new user prompts and feedback into the pipeline, fueling the flywheel.

2. Data Storage & Processing. Use databases, warehouses (Snowflake, Redshift), cloud storage (S3, GCS), and ETL/ELT processes. Consolidate raw logs, user conversation histories, and curated data into a central repository for later efficient querying and analysis.

3. Data Analysis & Insight. Employ advanced analytics and machine learning to spot patterns, anomalies, or valuable features. For LLM data, this might involve clustering prompts by topic, scoring response quality, or detecting harmful/bias issues.

4. Value Realization. Translate analytical findings into real‐world impact—such as refining recommendation engines or business processes. Crucially, feed these insights back to model‐training pipelines, whether that’s updating the training corpus, adjusting prompts, or refining fine‐tuning methods.

5. Feedback Loop Enhancement. The improvements and benefits brought by applying new data will further increase the quality and quantity of the next round of data. This may include better data collection methods, more accurate data annotation, etc.

Data Synthesis

• Prompt Synthesis:
• Self-Instruct Methodology (from the Self-Instruct paper): Apply task identification by tagging each task with task_type labels. Prepare seed prompts corresponding to each task_type. Randomly sample seed prompts and input them into a high-capacity model. Use the model to generate new diverse prompts from the seed prompts.
• Heuristic-Based Synthesis: Compile diverse datasets across multiple task_type categories. Perform rewriting of data through rule-based transformations and high-capacity LLMs. Generate variations in multiple formats (e.g., text, markdown, JSON) and diverse prompting styles.

• Answer Synthesis:
• Poweful model is all you need. For generating high-quality answers, powerful models (e.g. GPT-4) remain the gold standard. For English data or scenarios where cost is not a major concern, use the latest model of GPT, Claude, Genimi, or Grok. For Chinese data or cost-sensitive scenarios, use latest model of Deepseek, or deploy models locally, such as QwQ-32B or DeepSeek-V3-0324.
• Small models + SFT ≈ Powerful models + zero-shot/few-shot/chain-of-thought. Smaller models fine-tuned using Supervised Fine-Tuning (SFT) can achieve performance similar to powerful models in zero-shot/few-shot cases. However, complex instructions and reasoning tasks may still be challenging. For specialized task types, we can generate ~1k high-quality answers using the latest powerful model, then fine-tune a smaller model with these answers. Finally, the fine-tuned smaller model will be used to generate 10k+ additional synthetic answers.
• Specialized Data: includes datasets specifically designed for: Long-context texts, agent or function-calling interactions, Retrieval-Augmented Generation (RAG-SFT) tasks.

• Industry Practices: Use rejection sampling, generating multiple inference paths for each prompt using methods such as Best-of-N (BoN) / Worst-of-N (WoN). Then select the optimal responses via human annotators (manual evaluation) or automated verifiers (using separate models or heuristic methods). The selected responses can be used to create chosen/rejected pairs for training via Supervised Fine-Tuning (SFT) or preference-based techniques like Direct Preference Optimization (DPO).

• Sandbox Validation: A “sandbox” is a controlled research and development (R&D) environment in which we can systematically evaluate:
• Synthetic data quality: Are the prompts and answers coherent, diverse, and error‐free?
• Model interactions—Does a model trained on this data respond as expected to various prompts?
• Potential flaws or biases—Are there systematic errors, duplications, or domain gaps?
By iterating in a sandbox rather than immediately deploying the data (or the model), we minimize risk and cost. Once the sandbox experiments confirm the data’s quality and alignment, we can scale up to higher‐cost training runs or full production deployments.



Typical Sandbox Pipeline:
• Data Pool Creation - Single-OP Processing: Suppose you have N different data‐processing “operations” or “recipes” . Each operation can produce different slices of synthetic data, for instance, [0,33%], [33,67%], and [67,100%] of the full dataset. These “cost‐controlled” pools let you test data in smaller chunks first.
• Train Small “Test” Models: Rather than training a large model right away, you train lightweight (or smaller) models on each data pool. Run multiple mini‐trainings (e.g., “3N+1 Low‐cost Models”, where are N data operations or subsets you want to compare, you might train three 3 small models per data operation plus one extra baseline or reference model) so you can compare how different synthetic slices or recipes affect downstream metrics (accuracy, perplexity, etc.).
• Initial Analyses:
1. Importance Analysis: Rank each data pool/operation by how much it improves specific metrics or tasks.
2. Correlation Analysis: Check how improvements on certain metrics correlate across pools (e.g., if a dataset boosts summarization accuracy, does it also help Q&A?).
3. Duplication Analysis: Ensure you are not re‐using the same data under different operations, which could bias the results or artificially inflate metrics.
4. Diversity Analysis: Confirm that your data covers varied task types, domains, styles, and difficulty levels.
• Multi‐OP Combinations: After seeing which single‐operation pools work best, you can combine multiple top operations (e.g., ). Repeat the same set of analyses above to see if combining them yields further gains or inadvertently causes duplications and domain skew.
• Iterative Refinement: Based on these analyses, pick the best data recipes or subsets. Potentially re‐synthesize or refine prompts/answers in areas where the model still underperforms. The result is a curated, “battle‐tested” dataset that you have confidence in.
• Higher‐Cost Scaling: Once the sandbox experiments show strong results, move on to larger or more expensive training runs with bigger models. Because you have validated the data pipeline in a controlled sandbox environment, you can safely invest in heavier computing or full production usage.

1. https://github.com/vndee/llm-sandbox

2. https://arxiv.org/abs/2410.23074

3. https://github.com/apple/ToolSandbox

Data Quality Filtering

• IFD Filtering (Instruction-Following Difficulty)
• Core Concept: We want to measure how “difficult” it is for a language model to follow an instruction in order to produce an answer . This difficulty score helps filter out poor-quality or unhelpful training data. The key observation is that LLMs already know a lot from pretraining, so fine-tuning primarily teaches them how to respond to instructions rather than imparting new factual knowledge. Hence, samples that are truly challenging from an “instruction-following” perspective will be the most valuable.
• Two Different Loss Terms
• :
is the character in question string. Measures how hard it is to generate given the instruction . Here, the model sees first, then autoregressively predicts each token of .
• :
Measures how hard it is to generate without any instruction at all. The model just tries to produce from scratch (no as prior context). It sounds unusual at first because we typically think of LLMs as responding to a question or prompt. However, in language modeling research, it’s perfectly valid to measure the likelihood of a sequence with no preceding context—sometimes referred to as unconditional or free-form language modeling. In practice, this is done by feeding in a “null” or dummy prompt, often just a special beginning-of-sequence (BOS) token, and then having the model predict the tokens in one by one. Then computing the cross-entropy loss (or perplexity).
• Intuitively
• answers: “How challenging is if the model already knows the intended question or instruction?”
• answer: “How challenging is if the model has no idea what the question or context is?”
By comparing these two, we see how much harder (or easier) it becomes when the model must explicitly “follow” the instruction .
• The Instruction-Following Difficulty (IFD)
We define:
• If is large, it means generating is significantly more difficult when following than when generating freely—hence the instruction demands more from the model.
• If is small, the instruction does not add much extra challenge: the model can produce easily based on the instruction.
• Workflow:
❓
• Step 1 - Initial Learning Phase (Establish Baseline): We first train the model on a small but diverse subset of $$(Q,A)$$ data. To ensure diversity, we might cluster the instruction embeddings (e.g., via K-Means) into multiple clusters, then sample a few examples from each cluster. After fine-tuning the model on this subset for 1 epoch, it gains basic instruction-following competence.
Where:
: The character in question string of sample j
: Corresponding final hidden state
• Step 2 - Empirical Evaluation (Computing IFD): We run the (now instruction-aware) model over larger datasets. For each (Q, A) pair, we compute two losses and , then we form the ratio .
• Step 3 - Filtering Based on Difficulty: We remove (filter out) pairs for which is too low. This ensures we keep samples that are genuinely instructive for learning instruction-following rather than trivial samples.
By focusing on the ratio , we effectively measure instruction-following difficulty rather than mere knowledge recall. This approach yields a higher-quality fine-tuning set, helping the model improve precisely where it needs to: following instructions.

• MoDS Filtering: Primarily filters data through three metrics—quality evaluation, diverse data selection for seed instructions, and augmented data selection.



• Quality Evaluation: The OpenAssistant's reward-model-debertav3-large-v2 model as reward model is used to score the quality of the data. The Instruction, Input, and Output sections of the raw data are concatenated and fed into the reward model to obtain a score. If the score exceeds , the data is deemed high-quality and included in the refined dataset. The High-Quality Instruction Data is generated in this step
• Diverse Data Selection for Seed Instructions: The K-Center-Greedy algorithm is employed to previously selected High-Quality Instruction Data, maximizing diversity while minimizing the size of the seed instruction dataset. Then, the Seed-Instruction Data is generated in this step.



Key Steps:
• Inputs: Takes a dataset , an initial pool , and a budget .
• Selection Rule: In each iteration, the point farthest from the current subset (measured by a distance metric ) is added, i.e. Maximizing the minimum distance.
• Termination: Stops when the subset size reaches .
• Output: Returns the newly selected points , where is the set minus operator, and equal to all elements in that are not in .
• Augmented Data Selection: After obtaining the Seed Instruction Data (the diverse high-quality subset used to train an initial model), the Augmented Data Selection step addresses instructions on which this initial model performs poorly. The process is:
• Seed-Based Initial Training: Fine-tune a pre-trained model on the Seed Instruction Data. The result is an initial LLM specialized for those core instructions.
• Model Inference & Scoring: Run the initial LLM on all instructions in the High-Quality Instruction Data (the data generated in step 1). Use the reward model in step 1 again to score the new predictions. Any instruction whose predicted response scores below becomes part of the necessity dataset (Data2). These are instructions the initial model cannot handle well.
• Diversity Filtering: Apply the K-Center-Greedy (again) to the necessity dataset (Data2) to select a diverse set of these “difficult” instructions. The resulting subset forms the Augmented Instruction Data, which captures instruction types that still challenge the model.
Fine-Tuning With Selected Data: In the final stage, the Seed Instruction Data and Augmented Instruction Data are combined to train a refined model (the “final LLM”). By including both the diverse high-quality (seed) and the challenging instructions (augmented), the model achieves broader coverage and higher quality across varying instruction types.

Data Diversity Exploration

1. Data Usage (Task Type) Diversity
1. Leverage known tasks:
• OpenAI’s website provides a comprehensive list of tasks ChatGPT excels at—such as translation or emoji-based conversations. Use these as a starting point for task selection.
• As language models, LLMs should also handle traditional NLP tasks (e.g., Named Entity Recognition NER, machine reading comprehension, intent recognition).
2. Align with business needs: If your downstream applications require specific scenarios, incorporate them during supervised fine-tuning (SFT). For instance, before Chinese New Year, companies might train the model on writing Spring Festival couplets or solving lantern riddles—provided the data is high-quality. Such specialized data generally does not degrade the model’s overall capabilities.
3. Balancing task difficulty: Tasks often appear in a hierarchical structure, e.g., “Logical Reasoning – Common Sense Reasoning” vs. “Logical Reasoning – Chain-of-Thought (CoT) Multi-Step Reasoning”. When distributing data across task types, avoid giving each task the same amount of data or data egalitarianism. Instead, allocate more data to challenging tasks and less to simpler ones, and continuously adjust it based on your base model’s capabilities.

2. Data Format Diversity
1. Varied prompt expressions: Avoid using the same instruction repeatedly (e.g., always using “Translate Chinese sentence A to English”). Introduce prompts in different contexts, such as “I’m traveling in the UK and need to ask for directions. How do I say A in English?”, “I’m an English teacher helping students write more naturally. How should I explain sentence A to them?”. This is to ensure the model doesn’t overfit to specific prompt patterns or key tokens.
2. Prompt difficulty control: Keep prompts within the same semantic space but vary their complexity. For example, the Wizard method (an instruction-evolution approach) uses GPT-4 to incrementally increase prompt difficulty, providing a gradient of complexity levels.
3. Prompt length balance: Include both short and long prompts to prevent the model from neglecting longer contexts. For long prompts, place critical information in different parts (beginning, middle, or end) to discourage the model from focusing solely on start/end tokens.
4. Answer length balance: Include scenarios where the model must produce longer outputs (e.g., “write at least 2000 words”), instead of always offering short replies.
5. Multi-turn conversation topic switching: Mix queries that rely on session context with queries that introduce entirely new topics. This teaches the model to distinguish when context is relevant and when it is not.
6. Answer distribution diversity: Avoid training samples that yield identical answers across many prompts. Excessively homogeneous answers can lead to overfitting, as they disproportionately affect the model’s loss calculation.
Summary (Data Format): All these measures aim to minimize predictable patterns in prompt/answer structures and to distribute key information in prompts randomly. This helps prevent the model from learning superficial cues (e.g., fixating on certain tokens or prompt positions). Research on using “LLM-as-Judge” (using LLMs to score or rank responses) reinforces the importance of avoiding any kind of bias.

3. Semantic-Level Diversity
1. Sampling strategy trade-offs: Consider different sampling methods (e.g., top-N, linear, logarithmic) for data selection, and weigh their pros and cons.
2. Measuring diversity shifts in embeddings: Use methods such as Sentence-BERT (SBERT) to obtain sentence embeddings. Compare distribution changes (before and after sampling) in vector space. Track statistical metrics like average pairwise distances or token frequency to ensure semantic variety.
3. Embedding-based clustering: Apply K-Means or K-Nearest Neighbors (KNN) on SBERT/SimBERT embeddings to cluster samples. Assign diversity weights to each data point, ensuring a balanced representation of semantic variations.



This figure presents a t-SNE-based visualization of the Alpaca dataset’s instruction embeddings in 2D space, highlighting the “cherry data” distribution. Each point represents an instruction embedding in two-dimensional space, with:
• Red Points: the top 5% of samples by IFD (Instruction Difficulty) score.
• Cyan Points: the bottom 5% of samples by IFD score.
Notably, these data points do not scatter uniformly. Instead, they form distinct clusters that reflect different degrees of task complexity. For instance, regions dominated by low-IFD clusters tend to consist of simpler requests—like minor text edits—whereas high-IFD clusters often feature more elaborate tasks (e.g., storytelling or in-depth explanations). This distribution challenges the assumption that cherry data is evenly spread across the instruction space and underscores the importance of including diverse, more complex requests to fully engage a large language model’s latent capabilities.

Open Source Dataset

3.1.3 SFT Training

Training Framework

Parameter Settings

• Regardless of the chosen framework, the following parameters are critical. Always review them before each training session:
• epoch: Typically set to 1. For domain-specific fine-tuning with limited data (≤10k samples), you may increase up to 3 epochs. Some overfitting may be acceptable if it improves domain-specific performance.
• gradient_accumulation_steps: The number of mini-batches for which gradients are accumulated before parameters are updated. The effective global batch size is:
• global_batch_size (Megatron's parameter; DeepSpeed users can typically ignore): Total batch size across all devices during training. Often derived from per_device_batch_size, num_devices, and gradient_accumulation_steps.
Note: Under OpenRLHF with DeepSpeed, you will find similar settings:
• learning_rate: In SFT, the lr (learning rate) is typically ~10× higher than in pretraining. For example, if pretraining used , an SFT lr might be .
• lr_scheduler_type: Common choices include constant, linear, cosine, and exponential. cosine scheduling is widely used.
• dropout: Often avoided in SFT because it typically provides minimal benefit and can reduce training efficiency.

• Key Parameters Affecting Training Speed
• zero_stage: Refers to DeepSpeed’s different Zero Redundancy Optimization (ZeRO) stages. zero2 is recommended if you have sufficient GPU memory; zero3 can slow training due to higher communication overhead.
• max_seq_len: Maximum input sequence length, commonly 4k (4096 tokens).
• offload: Offloads some GPU data or computation to CPU/NVMe. Rarely used in most SFT scenarios unless GPU memory is highly constrained.
• gradient_checkpointing: Discards intermediate activations during the forward pass, then recomputes them for the backward pass. Significantly reduces GPU memory usage at the cost of increased compute time (roughly 2× the forward pass).
🎓
Gradient Checkpointing in PyTorch:
PyTorch implements gradient checkpointing via torch.utils.checkpoint.checkpoint and torch.utils.checkpoint.checkpoint_sequential:
• Forward pass: Saves only the necessary input tuples for each function (instead of all intermediate activations).
• Backward pass: Recomputes the forward pass on-the-fly to obtain gradients, then discards intermediate data again.
This approach nearly doubles forward-pass computation time but reduces GPU memory usage considerably. It reduces dynamic memory usage from (where is the number of layers) to .
• seq_parallel_size: Related to sequence parallelism. Splits the input sequence into smaller segments for distributed processing.

• Other Parameters (Lower Impact, but Worth Noting):
• weight_decay: Penalizing large weights; default is often 0.01.
• per_device_train_batch_size: The batch size allocated per GPU. A value of 1 is common in large-model SFT, especially if memory is a concern.
• num_warmup_steps: Number of steps for learning-rate warmup. Usually 5%–10% of the total training steps (e.g., 500–1,000 steps in a 10,000-step run).

Training Techniques

• Channel-Specific Loss Monitoring: Different task_type may produce different loss behaviors. For clearer diagnostics, track each channel_loss separately, where different channels indicate different tasks.

• Special Token Loss: Loss on special tokens (e.g., <bos>, <eos>, <pad>) may start relatively high but usually drops quickly.

• Creative vs. Fixed-Answer Tasks: Creative tasks often have inherently higher loss because there are many equally valid responses. Fixed-answer or deterministic tasks (where the “correct” answer is narrowly defined) typically show lower loss.

• Expected Loss Range by Model Size: For 7B/13B parameter models, with well-sampled general-purpose data, an initial loss around 2 (or up to 3 for harder data) is common. 72B models often have initial losses between 1 and 2. The final loss typically converges to about 0.5, depending on different models. If it goes much lower, the model might be overfitting—leading to near-zero probability to tokens not in its training distribution.

• Diagnosing Increasing Loss: If the loss consistently increases during training, do not doubt the data. The next-token-prediction (language modeling) paradigm is essentially memorization and can always lead to overfitting and very low losses on training data, so “failing to learn” usually indicates a training code issue. Even with random noise as input (no meaningful pattern), the loss should plateau rather than rise indefinitely.

Training Strategies

1. Multi-task Learning: Combine multiple SFT data sources (e.g., coding, math, and general capability) into a single, mixed dataset. If each data source is considered a single task, this approach effectively becomes multi-task learning.

2. Sequential Training: Apply SFT to each dataset in sequence. For example, train on coding data first. Then train on mathematical reasoning data. Finally, train on general capability data. Each stage “builds on” the model resulting from the previous stage.

3. Mixed Sequential Training:
1. Stage 1: Multi-task learning on specialized datasets (e.g., code and math) merged into a single set.
2. Stage 2: Fine-tune on a general capability dataset.
This approach first focuses on specialized skills, then consolidates them with a broader capability set.

4. Dual-stage Mixed Fine-tuning (DMT)
1. Stage 1: SFT on specialized datasets (same as the first stage of Mixed Sequential Training).
2. Stage 2: SFT on a blended dataset , and . It includes general data plus different proportions of code/math data (e.g., )
Retaining some code/math data in the second stage helps the model preserve (or “recall”) its specialized capabilities while learning from general data.

Multi-Turn Dialogue Enhancement

1. Multi-Turn Dialogue Data Detection:
1. Collect Multi-Turn Conversation: Gather user-bot dialogue turns from online logs. A small discriminator model is then trained to label each turn pair:
• 1 for “continuous” (the conversation remains on topic),
• 0 for “discontinuous” (the conversation shifts to a new topic).
2. Data Selection:
• Real multi-turn data (with a consistent theme across turns) is added directly to the training set.
• Pseudo multi-turn data (with noticeable topic shifts) is selectively included in small quantities. Including pseudo multi-turn data increases the model’s learning difficulty, ultimately improving its ability to handle sudden context changes and preventing “focus drift” over long contexts.
Example: A pseudo multi-turn dialogue might start with questions about a product, then abruptly shift to discussing the weather. Training on such examples teaches the model to discard obsolete context and re-focus on the new topic when necessary.

2. Multi-Turn Dialogue Data Synthesis: You can synthesize longer dialogues from single-turn samples. For instance, take an existing single-turn dialog sample, and construct a new prompt for the next round via templates or GPT-based generation. Then continue generating the response.

3. Multi-Turn Dialogue Data Flywheel: Continuously collect online logs, especially recent multi-turn conversations. These fresh dialogues can then be incorporated into your training set to keep the model updated with the latest user behaviors and topics.

4. Multi-Turn Loss Calculation:
1. Multi-Turn Sample Splitting: Suppose we have a conversation with three user-bot turns. We can split it into three separate samples. Each sample includes the prompt (user input plus previous messages) and the bot’s current response. In SFT, only the bot’s current output tokens are used for calculating loss (i.e., previous tokens are masked with -100).



If is the total loss for sample , and is the number of output tokens in that sample, a straightforward approach is to use:
This method handles length normalization properly but slows the training because prefixes (e.g. Sytem + User1) are repeatedly processed separately.
2. Multi-Turn Merging for Accelerated Computation: To save computation, you can merge multiple turns into one long sample. Because of the causal attention mask, each token only sees the preceding tokens, so in principle it is “equivalent” to splitting.



However, there is a pitfall involving loss reduction. By default, PyTorch’s CrossEntropyLoss with reduction="mean" does:
Hence, we end up with:
When the three turns have different output lengths, this effectively underweights shorter responses and overweights longer ones, resulting in under-training of short-response data.
3. Packing + Accelerated Computation: Consider two dialogues: one single-turn and one three-turn.



The correct loss should be:
Using packing, all samples are concatenated, with attention masks ensuring tokens from later samples cannot attend to earlier ones. For example, in FlashAttention, this can be implemented using flash_attn_varlen_qkypacked_func with the cu_seqlens parameter. However, similar to the previous case, without modifying the loss calculation method, packing samples with varying lengths will still lead to uneven training.
4. Final Solution: The goal is to disable the default averaging across tokens so that we can explicitly compute the per-turn loss. In many training frameworks (e.g., Megatron-LM), the following three averaging steps typically occur:
1. Micro-batch dimension: the average is taken over unmasked tokens within each micro-batch
2. Data-parallel (DP) dimension: averaging across multiple GPUs
3. Gradient-accumulation dimension: averaging across multiple accumulation steps
We want a custom loss that normalizes each turn’s loss by that turn’s token count, and then sums over turns, finally dividing by the total number of turns in the batch. In newer versions of Megatron-LM, setting the flag --calculate-per-token-loss disables the data-parallel and gradient-accumulation averaging. Then provide a custom loss_func that returns:
• A total scaler loss for backpropagation, which is the sum of per-token loss for different samples in micro-batch, which works for both packing or non-packing sample
• The number of dialogue turns in the batch (used as the denominator for the optimizer to scale the gradients)
• Logging info (e.g., total loss sum, total turn count)
Below is an example implementation that shows how to incorporate per-turn normalization:

Training Launch Script

3.1.4 SFT Evaluation

1. Automated Evaluation: When using GPT-4 or Claude for evaluation, prompt design must be handled carefully. Large language models often exhibit biases—for example, they may favor option A over B between similar A and B, or favor longer answers over short answers. In scoring tasks, even correct answers can receive inconsistent evaluations (e.g., 3, 4, or 5 across separate trials). The automated evaluation prompt from Alignbench can be a useful reference. Besides, including a reference answer can also improve scoring consistency by allowing the model to compare the “candidate answer” against the “gold answer.”

2. Human Evaluation: This is quite self-explained. Humans review the responses and provide ratings. Although more labor-intensive, human evaluation often yields judgments that are less prone to the systematic biases found in LLMs.