AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
11.7 SigLIP
Limitations of CLIP
CLIP uses image-text contrastive loss for training networks. This method has several advantages:
- By crawling the internet data, the cost of collecting image-text data is relatively low.
- Supports zero-shot transfer to downstream tasks (such as image classification/retrieval).
- Performance scales with model and dataset expansion, meaning larger networks and datasets bring better performance.
However, CLIP faces two technical challenges:
- It requires large-batch training. For example, CLIP uses a batch size of 32K, which demands a large number of GPUs.
- It requires extensive communication between these GPUs. Specifically, image and text features need to perform all-gather operations across all GPUs. Considering the required large batch sizes, this type of communication is very extensive.
CLIP uses contrastive learning, where for each image-text pair, positive and negative samples need to be distinguished within the batch.
This is not a problem during single-machine training. For example, if the batch size is
256, then a 256x256 similarity matrix is constructed.However, during multi-GPU training, each GPU might only have a batch size of
64, meaning you only have local positive and negative pairs.To achieve global contrast (the entire batch size
64x4=256), CLIP performs the following operations:- All-gather image features: Each GPU sends its own image features to all other GPUs;
- All-gather text features: Similarly, all GPUs broadcast their own text features;
- Concatenate into a complete large batch: This allows for the calculation of a global similarity matrix;
- Then calculate InfoNCE-style contrastive loss.
Loss Functions of SigLIP and CLIP
SigLIP reduces CLIP's demand for memory and communication. SigLIP is an abbreviation for Sigmoid Language Image Pre-training. The core idea of SigLIP is to use the sigmoid operation instead of the softmax operation.
CLIP uses a softmax function, so the loss for a given positive sample (image-text) depends on every negative sample in the batch, as shown below.
- The CLIP (using softmax) loss is asymmetric and contains two terms. The first term finds the best matching text for a given query image, while the second term finds the best matching image for a given query text.
- The CLIP loss requires a global normalization factor (the denominator).
In contrast, SigLIP is neither asymmetric nor does it require a global normalization factor. The loss for each sample pair (positive or negative pair) is independent of other pairs in the batch, as shown below.
where is the label for a given image and text input, which equals 1 if they are paired and -1 otherwise.
Obviously, it does not require maintaining a global
N×N matrix for calculating the normalization factor used in CLIP. Therefore, SigLiP's loss can be calculated incrementally, which is suitable for large-batch training.Note that it still requires the all-gather operation to transfer features to all other GPUs; but instead of transfering all image features and text features, because the loss is symmetric, only transfering text features will suffice, which largely reducing the communication burden
Efficient “chunked” implementation
SigLIP has lower inter-GPU communication costs than CLIP. CLIP requires two all-gather operations (for image and text features) to compute an similarity matrix. In contrast, SigLIP only needs to gather text features, requiring just one all-gather.
Even so, one all-gather is still costly. All GPUs must wait to receive features from others before computing the loss. For example, with 256 GPUs, each must wait for features from 255 others.
To avoid this, SigLIP uses a "chunking" strategy, eliminating the need for all-gather. Since SigLIP’s loss is computed pairwise and independently, the loss can be calculated incrementally.
As shown in the above figure (with 3 GPUs and batch size 12):
- Each GPU first computes the loss for its local batch.
- Then, it sends text features to a neighboring GPU.
- With each step, GPUs receive new negatives and add to their loss.
This loop of local loss + minimal feature exchange continues until the full batch loss is accumulated. This is more efficient and without needing a global all-gather.
Experiments
In the experimental section, SigLiP (sigmoid) and CLIP (softmax) were comparatively evaluated.
In the middle plot:
- In small batch sizes (e.g., 4-8K), SigLiP's performance surpasses CLIP's; this is very important because many researchers lack the computational resources (GPUs) for large-batch training.
- Although related work claims that large batches can improve performance, this paper shows that both SigLiP and CLIP reach saturation at a 32K batch size.
- As the batch size increases, the performance gap between SigLiP and CLIP gradually narrows.
As for the SigLiT
- SigLiT (Sigmoid LiT) is a model that combines the LiT (Locked-image Tuning) method with Sigmoid loss.
The characteristic of LiT is that the image model is pre-trained and frozen (not updated), only the text encoder is trained for image-text matching.
- SigLiT uses sigmoid loss instead of the traditional softmax contrastive loss, which allows it to better adapt to small batch sizes and be more memory-efficient.
- The ViT-g/14 model can achieve 84.5% ImageNet zero-shot accuracy using the loss provided in SigLIP.
mSigLiP (multilingual SigLiP)
- mSigLiP is a multilingual extended version of SigLiP, used to process image-text data in over 100 languages.
- Even in multilingual scenarios, a reasonable batch size (e.g., 32k) is sufficient; further increasing it can actually lead to performance degradation.
Prev
11.6 BLIP-2
Next
Large Language Models Pass the Turing Test.
Loading...