7.3 Knowledge Distillation

7.3.1 Introduction

• Blackbox knowledge distillation, where typically only the teacher model’s outputs can be accessed—these teacher models often come from closed-source large language models; and

• Whitebox knowledge distillation, where the teacher model’s parameters and output distributions are accessible—these teacher models are usually open-source large language models.

7.3.2 White-box Knowledge Distillation

• Response-based: When distilling this type of knowledge, the student model directly learns the output-layer features of the teacher model. Simply put, the teacher fully learns the knowledge first, then directly tells the student the final conclusion.

• Feature-based: The previous method involves a very direct learning goal—the student model learns the final prediction from the teacher model. Considering that deep neural networks learn hierarchical features, the intermediate feature activations of the teacher model can also serve as learning targets for the student model, complementing response-based knowledge.
Although feature-based knowledge transfer provides more information to the student, the structures of teacher and student models often differ. For example, determining which layer’s features from the teacher model (hint layer) match with which layer of the student model (guided layer) is still an open research question.
In addition, differences in size between hint and guided layers pose challenges for effective feature matching, and there is currently no mature solution.

• Relation-based: The above two methods rely on outputs from specific layers of the teacher model. In contrast, relation-based knowledge goes a step further by exploring the relationships between outputs of different network layers or between different samples.
For example, it may use the Gram matrix of feature maps from two layers in the teacher model (representing inter-layer relationships), or the probability distribution over feature representations of different samples in the teacher model (representing inter-sample relationships) as the knowledge to be transferred.

The Gram matrix is a famous loss function used for style transfer a few years ago in this paper: A Neural Algorithm of Artistic Style

• Offline distillation: The most widely used method in knowledge distillation. It mainly includes two steps:
• Pre-train the teacher model
• Transfer knowledge through distillation.
This approach focuses on knowledge transfer. The teacher model usually has a large number of parameters and a long training time. Some large models use this method to produce smaller models — for example, BERT distilled into tinyBERT.

• Online distillation: Requires the teacher model and student model to be updated simultaneously. This method is typically used when the teacher model with high accuracy is not easily obtainable beforehand.

• Self-distillation: A special case of online distillation where the teacher and student models share the same network architecture. Used to improve the generalization capability and performance of the model.

7.3.3 Black-box Knowledge Distillation

Note that even smaller LLMs can have a certain in-context learning and CoT abilities, so we can do these in a zero-shot behavior. But below works focus on training the student LLMs to further enhance their ICL, CoT, and IF abilities.

• ICL: In-context learning uses structured natural language prompts that include task descriptions and examples. Through these task examples, LLMs can generalize to new tasks without explicit fine-tuning.
ICL distillation transfers LLMs’ few-shot and in-context learning abilities to student models.

• CoT: Compared to ICL, Chain-of-Thought introduces intermediate reasoning steps. These steps can influence the final output, rather than relying solely on a simple input-output mapping. Some CoT-based distillation methods include:
• MT-CoT, which enhances smaller reasoning models by training on explanations generated by larger LLMs via multi-task frameworks, helps the student model acquire logical and commonsense reasoning abilities.
• Fine-tune-CoT, a method that generates multiple reasoning samples from very large teacher models to fine-tune smaller models.
• SOCRATIC CoT trains two distilled models (problem composer and subproblem solver), breaking complex questions into sub-questions and solving them separately.
• DISCO: An automated method for generating high-quality counterfactual data using LLMs. It prompts LLMs to create perturbed sentences and uses task-specific teacher models to filter them. The filtered data is then used to train student models, improving their robustness and generalization in counterfactual reasoning tasks.
• SCOTT, a method that improves reasoning in language models by using contrastive decoding to link each rationale with the correct answer, encouraging the teacher model to generate more relevant explanations. It also trains student models with both correct and incorrect rationales to improve faithfulness and avoid shallow reasoning.

• IF: Instruction Following relies on task descriptions with little or no examples. By using a series of task instructions in natural language, language models demonstrate their ability to generalize to unseen tasks.
Instruction-following distillation transfers this capability to student models by leveraging LLMs’ adaptability—prompting LLMs to generate “hard” instructions and then using these to enhance the student’s ability to follow complex instructions.