4.7 DeepSeek Series

4.7 Deepseek Series

4.7.1 Deepseek-V1

• Model Architecture:
• Based on LLaMA, incorporating Pre-RMSNorm, SwiGLU, and RoPE. The 67B model uses GQA (Grouped Query Attention) to reduce inference cost, and the number of layers is adjusted accordingly.
• Uses BBPE (Byte-Level BPE) for tokenization. Trained on approximately 24GB of corpus, with a vocabulary size of 102,400.

4.7.2 Deepseek-math

4.7.3 Deepseek-V2

• Model Structure:
• Utilizes MLA (Multi-Head Latent Attention).
• MoE (Mixture-of-Experts): Limits the number of active experts to within M devices. Specifically, the process involves first selecting the M devices with the highest scores and then choosing the top-k experts among these M devices.
• Communication Balance Loss: Although Device-Limited Routing addresses the issue of uneven data distributing among devices, there can still be scenarios where a minority of devices receive most data, causing communication bottlenecks. Thus, DeepSeek-V2 introduces an additional Communication Balance Loss, as the equations shown below.
Here, D represents the total number of devices, and M is the number of devices limited by routing. The ratio ensures the loss has minimal dependency on the number of devices or the activated experts per device. Device-Limited Routing guarantees that each device transmits at most MT hidden states, while Communication Balance Loss ensures each device receives approximately MT hidden states, thus achieving balanced load across devices.
• Token-Dropping Strategy: Despite the previous efforts to achieve load balancing, complete balance is still not guaranteed. Therefore, DeepSeek-V2 introduces a device-level token-dropping strategy. Firstly, it estimates the average computational budget per device. Then, tokens allocated to each device are sorted in descending order based on routing scores. Finally, tokens exceeding the average computational budget per device are dropped from the end, excluding them from hidden states computation in that layer.

• Model Training:
• Multi-Head latent Attention (MLA): low-rank key-value joint compression. For details, check Section 1.3.5
• GRPO Algorithm: Group Relative Policy Optimization. For details, check Section 3.4.1

4.7.4 Deepseek-V3

• Model Architecture: Primarily focuses on MoE structural improvement. Compared with traditional MoE architectures, DeepSeekMoE adopts a finer-grained expert routing mechanism, and innovatively assigns some experts as shared experts. DeepSeek-V3 introduces an auxiliary-loss-free strategy for load balancing, Complementary Sequence-Wise Auxiliary Loss and Node-Limited Routing to ensure almost complete parallelism in computation and communication, and can process the entire input sequence of tokens without dropping during both training and inference.



• Gating computation changed from softmax to sigmoid: This can be seen as changing from multi-class classification (one data to one category among multiple categories) to multi-label classification (one data to more than one category among multiple categories).
• Reason: from v2 to v3, the number of routing experts increased from 160 to 256. However, softmax is a saturating activation function that tends to push output probabilities toward the extremes of 0 or 1. This causes most experts on the routing path to receive very small scores, leading to poor utilization and discrimination of experts. As the number of experts increases, this issue becomes more severe. But replacing the overall softmax with per-expert sigmoid activations, the interactions between different experts at the gate layer are reduced, which also weakens the "competition mechanism" among them.
• Auxiliary-Loss-Free Load Balancing: In DeepSeek-V1 and V2, auxiliary losses were introduced at the expert level, device level, and communication level to address load imbalance. However, these auxiliary losses often degraded model performance. Therefore, this paper removes such auxiliary losses, and instead introduces a learnable bias term to control Top-k routing, ensuring load balance.
• Specifically, the previous method used the expert scores directly for Top-k selection. The new approach adds a learnable bias term to the expert score before Top-k selection. If an expert is overloaded, its bias is reduced, decreasing its chance of being selected. If an expert is underutilized, its bias is increased, raising its probability of activation.
• Complementary Sequence-Wise Auxiliary Loss: DeepSeek-V3 also introduces an auxiliary loss at the sequence level. The idea is essentially the same as the previous expert-level and device-level auxiliary losses — the only difference is that it operates at a finer granularity (i.e., per sequence).
• No Token-Dropping: Thanks to the newly introduced techniques achieving better load balancing, the previously used token-dropping trick is no longer necessary and can be removed.

• Training Objective: The training objective is Multi-token Prediction (MTP). DeepSeek-V3 adopts the MTP objective by expanding the prediction scope to multiple future tokens at each position. This design has two major advantages:
1. By increasing the density of training signals, MTP improves data utilization efficiency.
2. It enables the model to plan ahead and predict future tokens more accurately.
Compared to previous MTP approaches, which predict D extra tokens using an independent output head, DeepSeek-V3 adopts a sequential prediction strategy, maintaining the causal chain of predictions across all prediction layers.

4.7.5 Deepseek-R1