Lazy loaded image11.3 BLIP

Paper: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
notion image
Bootstrapping Language-Image Pre-training (BLIP) is a vision-language understanding and generation pre-training model. Its core idea is to generate high-quality text annotations through the Bootstrapping method and the Captioner-Filter mechanism, thereby improving the quality and quantity of pre-training data.
BLIP achieves alignment and generation of vision and language through the Multimodal mixture of Encoder-Decoder (MED) structure.
Bootstrapping is a statistical estimation method that resamples observed data to infer the distribution of the population. In the BLIP, it’s a bit different and used for augmenting original datasets by generating and filtering new data. Compare it with the Bootstrapping in Reinforcement Learning.

MED Architecture

The core of BLIP is the Multimodal mixture of Encoder-Decoder (MED) architecture, including the following four modules:
Image Encoder
  • Based on the ViT architecture.
    • The input image is divided into multiple patches, and encoded as a series of Image Embeddings.
    • Uses the [CLS] token to represent global image features.
  • Function: Extracts image features, used for contrastive learning (similar to CLIP's Image Encoder).
Text Encoder
  • Based on the BERT architecture.
  • Adds the [CLS] token at the beginning of the input text to represent global text features.
  • Function: Extracts text features, used for contrastive learning (similar to CLIP's Text Encoder).
Image-grounded Text Encoder
  • Cross-attention layers are used between the self-attention layer and the feed-forward network in the Text Encoder to inject visual information.
  • Adds an [Encode] token at the beginning of the input text to mark the encoding task; it’s also used as the multimodal representation of the image-text pair.
  • Function: Extracts text features and aligns them with image features (more fine-grained Text-Image alignment than CLIP).
Image-grounded Text Decoder
  • Replaces the self-attention layer of the Image-grounded Text Encoder with a Causal Self-Attention layer.
  • Adds [Decode] and [EOS] tokens at the beginning and end of the input text, respectively, marking the start and end of the sequence.
  • Function: Generates text descriptions grounded in image and text features (CLIP does not have this capability).
Note that the above four modules share same parameters among them, as indicated by the same colors in the workflow shown in the beginning.

Loss Functions

BLIP jointly optimizes three objective functions during pre-training. All are common loss functions in contrastive learning like ALBEF.
Image-Text Contrastive (ITC) Loss
  • Goal: Align image and text in the feature space (just like CLIP).
  • Method:
    • Maximize the similarity between positive image-text pairs.
    • Minimize the similarity between negative image-text pairs.
    • Uses soft labels generated by the momentum encoder for auxiliary training.
        • This is from the ALBEF. This technique can handle the potential positive matching in the negative pairs. For example, some other texts in the same batch still contain certain correct infomation in the image.
        The soft labels in ALBEF
        The soft labels in ALBEF
  • Purpose: Used to train the Image Encoder and Text Encoder.
Image-Text Matching (ITM) Loss
  • Goal: Achieve fine-grained alignment between vision and language.
  • Method:
    • Predict whether an image-text pair is a positive or negative sample through a binary classification task (so an ITM head is added for classification).
    • Uses hard negative mining techniques to effectively capture negative sample information.
        • This is also from ALBEF. “A negative image-text pair is hard if they share similar semantics but differ in fine-grained details”. The ones within the same batch but are more similar to the image/text are harder negatives, so a multinomial distribution is constructed and sampled to get hard negatives according to their similarities to the image/text.
  • Purpose: Used to train the Image-grounded Text Encoder.
Language Modeling (LM) Loss
  • Goal: Generate text descriptions of images.
  • Method:
    • Trains the model to maximize text probability using an auto-regressive approach by optimizing the cross-entropy loss function.
    • Uses label smoothing to calculate the loss. For each class , the label becomes:
        • Where:
      • : Smoothed label for class i
      • y: True label index
      • : Smoothing factor (hyperparameter)
  • Purpose: Used to train the Image-grounded Text Decoder.

Captioner-Filter (CapFilt) Mechanism

Due to the limited availability of high-quality human-annotated datasets (such as the COCO dataset), BLIP collected a large number of image-text pairs from the web for training data.
However, the text in these web-based datasets typically contains noise. To address this, BLIP proposed the Captioning and Filtering (CapFilt) mechanism.
notion image
Captioner
  • Function: Based on the Image-grounded Text Decoder, it generates text descriptions for given images.
  • Training: Fine-tuned on the COCO dataset using the LM loss function, so no input text is needed for captioning.
  • Output: Given a web image , it generates a caption .
Filter
  • Function: Based on the Image-grounded Text Encoder, it removes noise from the text.
  • Training: Fine-tuned on the COCO dataset using the ITC and ITM loss functions.
  • Method: By comparing the matching similarity between the text and the image, it prunes the noise in the original Web text and synthesized text . A text is considered noisy and is removed if the ITM head predicts it as unmatched to the image.
Dataset Construction
Combines the filtered image-text pairs with human annotations to form a new, high-quality dataset, then uses this new dataset to retrain the BLIP model.

Training Process

  1. Initial Training: Train BLIP using noisy web data.
  1. Fine-tuning: Fine-tune the Captioner and Filter on the COCO dataset.
  1. Data Captioning: Use the Captioner to generate synthetic captions for web images.
  1. Data Filtering: Use the Filter to remove noise in the original web text and synthesized text, obtaining clean data.
  1. Final Training: Retrain BLIP using the clean data to obtain a high-performance model.
Prev
11.2 CLIP
Next
11.4 EVA
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant