AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
11.2 CLIP
Contrastive Language-Image Pre-Training (CLIP) is a pre-trained model released by OpenAI in 2021, used for matching images and text. It is a classic work in the field of multimodal research.
Model Structure
The CLIP model consists of two main parts:
- Text Encoder: Used for encoding text to generate text embeddings. It employs the classic text Transformer model.
- Image Encoder: Used for encoding images to generate image embeddings. The authors experimented with ResNet and Vision Transformer.
The embeddings generated by these two encoders are both single vectors of fixed length.
Model Training
CLIP's training is based on Contrastive Learning. The specific steps are as follows:
- Data Preparation: Assume a batch contains 64 text-image pairs, where each pair includes one text and one image.
- Encoding: Use the Text Encoder and Image Encoder to generate embeddings for text and images, respectively, obtaining:
visual_embedding:[64, embedding_size]text_embedding:[64, embedding_size]
- Positive and Negative Samples:
- For each text-image pair, the paired text and image are positive samples.
- For each text, the other 63 images in the same batch are negative samples.
- For each image, the other 63 texts in the same batch are negative samples.
- Data Augmentation:
- “A random square crop from resized images is the only data augmentation used during training.”
Note that in CLIP, there is no typical data augmentation used like SimCLR, MoCo to construct additional positive pairs (such as two different enhancements to the same image), because their main goal is cross-modal contrast learning (image-text), not image-image, and they believe that using real (image-text) pairs is enough to learn strong generalization representations under web-scale data.
Web-scale is a widely used term that usually means a large scale amount of data at the Internet level.
- Similarity Calculation: Perform cross-product between
visual_embeddingandtext_embeddingto obtain a[64, 64]matrix after feature normalization. The values on the diagonal of the matrix represent the inner product of the corresponding image-text features, indicating the cosine similarity of positive samples.
- Loss Calculation: For each row (i.e., the similarity between an image and all texts), set the label for the positive sample to 1. Training is performed using the cross-entropy loss function, with the goal of maximizing the inner product of positive samples.
You can think of the InfoNCE as cross-entropy applied to similarity logits, rather than model-generated class logits.
Code Example
Prev
11.1 Vision Transformer
Next
11.3 BLIP
Loading...