AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
7.2 Pruning
7.2.1 Introduction
- Model quantization refers to compressing a model by reducing the number of bits of weights or activations required for representation
- Model pruning studies redundant weights in the model, attempting to remove or trim unnecessary and non-critical weights to reduce model size and computational complexity
.png?table=block&id=1ef26e5a-7de0-8078-97ff-f5e6ee1d6fd7&t=1ef26e5a-7de0-8078-97ff-f5e6ee1d6fd7)
7.2.2 Pruning Workflows
There are three main approaches to model pruning:
- Train the model first, then prune, finally fine-tune
- Prune during model training, then fine-tune
- Prune the model, then retrain the pruned model from scratch
<ins/>
7.2.3 Types of Pruning
Unstructured Pruning
- Randomly removes individual weights or neuron connections, eliminating parameters without considering the overall network structure, or
- Prunes parameters below a certain threshold, which can lead to sparse parameter matrices with irregular structures.
As a result, it requires specially designed compression formats for storage and computation. Unstructured pruning often requires extensive retraining to restore accuracy, making it costly for LLMs.
- Pros: Efficient pruning algorithms; high compression ratio for the model
- Cons:
- Accuracy may be unstable;
- Sparse weight matrices are irregular, making it hard for hardware to effectively leverage compression and acceleration
Example of Unstructured Pruning Method:
- SparseGPT: Introduces a one-time pruning strategy that requires no retraining. It formulates pruning as a general sparsity regression problem and solves it using approximate sparsity regression solvers, achieving significant unstructured sparsity.
- LoRAPrune: Combines Parameter-Efficient Fine-Tuning (PEFT) methods with pruning to enhance downstream task performance. It introduces a unique parameter importance criterion based on LoRA values and gradients.
- Wanda: Unlike traditional pruning techniques that rely solely on weight magnitudes or require expensive weight updates (e.g., SparseGPT), Wanda computes weight importance using the product of the weight magnitude and the norm of the corresponding input activation, evaluated per output neuron. It requires no retraining, no weight updates, and can be done in a single forward pass.
Structured Pruning
Removes connections or layer structures based on predefined rules while preserving the overall network architecture. This method prunes groups of weights at once, offering advantages in reducing model complexity and memory usage while maintaining the full structure of LLMs.
Structured Pruning Method:
- LLM-Pruner: Uses a versatile approach to compress LLMs while preserving their multi-task reasoning and language generation capabilities. It introduces a dependency-aware scoring metric and identifies interdependencies within the model structure. It also implements an efficient importance estimation method that considers first-order information and approximated Hessian information.
Prev
7.1 Quantization
Next
7.3 Knowledge Distillation
Loading...