AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
9.3 o1 Techinical Paper
Outlines
This section focuses on research conducted primarily after the release of OpenAI-o1 and before DeepSeek-R1, during which the community actively speculated about, replicated, and laid foundational work related to the o1 technical roadmap.
- For logical reasoning tasks of low to moderate difficulty: Increasing computational effort at inference time (like tree search) can yield significant performance gains, often more effectively than improving reasoning abilities during pretraining.
- For more complex reasoning tasks: However, relying solely on inference-time computation tends to offer limited improvement and may even have bad effects. In such cases, it is more effective to strengthen the model’s reasoning capability during the pretraining phase.
Learning Strategy | Advantages | Disadvantages | Examples |
Behaviour Clone Expert | 1. More human-like, expert-like, and can reflect personal preferences
2. Can train through single-agent demonstrations
3. Performs well when data is abundant | 1. Real capability limited by biased data distribution
2. Cannot explore behaviors or ability outside human actions
3. Cannot utilize incorrect data | Game-playing AIs, LLM SFT |
RLHF | 1. Can align with human preferences and values
2. Can leverage incorrect data
3. High data efficiency | 1. Difficult to model preferences accurately and prone to reward hacking
2. High training cost | ChatGPT |
Self-play | 1. Maximum competitiveness, sometimes surpassing top human experts
2. Enables optimal strategy via zero-sum competition | 1. May fail to align with human behavior and actions appear non-human
2. Extremely high cost for training and inference | AlphaGo, OpenAI o1 |
Why does RL scaling?
The scaling benefits of pretraining have already been fully exploited, and the major remaining gains must now come from post-training.
o1 has shown that in certain domains, post-training still yields benefits, but achieving those gains through token-level supervision in SFT alone is no longer sufficient. Even scaling during training isn't enough anymore, the scaling during reasoning (inference) is also necessary.
That’s why researchers are now turning to scaling in the RL phase as a key direction.
Prev
9.2 MedLLM
Next
9.4 Prompting
Loading...