11.6 BLIP-2

Overview

• Image Encoder: Extracts visual features from the input image. This paper uses two different network architectures: ViT-L/14 trained in CLIP and ViT-g/14 trained in EVA-CLIP.

• Large Language Model (LLM): A large language model that generates text. This study attempts two different network architectures: decoder-based LLM and encoder-decoder-based LLM.

• Q-Former: Bridges the gap between visual and language modalities, facilitating cross-modal interaction and alignment. Q-Former uses a set of learnable query vectors (Queries) to extract visual features from the frozen Image Encoder, which are then fed into the frozen LLM to generate text. The Q-Former architecture consists of two modular components: Image Transformer and Text Transformer, both sharing a self-attention layer:
• Image Transformer: Used for interacting with the frozen image encoder, and extracting a fixed number of output features.
• Text Transformer: Served as both a text encoder and a text decoder, depending on the learning objective.

1. Image Encoder receives the image as input and outputs the visual features of the image.

2. Q-Former receives the visual features output by the Image Encoder and outputs a visual representation understood by the LLM.

3. LLM model receives the visual features output by Q-Former and generates corresponding text.

First-stage Pre-training: Representation Learning

• Calculate the similarity between the Query embedding from the Image Transformer and the text embedding from the Text Transformer.

• To avoid information leakage, ITC uses an unimodal self-attention mask, prohibiting direct interaction between Query and Text.

• The text embedding in the Text Transformer is the output embedding of the [CLS] token, while the Query embedding contains multiple output embeddings. Calculate the similarity of each Query embedding with the text embedding, and select the highest one as the image-text similarity.

• Due to the Q-Former's architecture not allowing direct interaction between the frozen image encoder and text tokens, the information needed for text generation must be extracted by the learnable Query and passed to the text tokens via a causal attention layer.

• ITG uses a multimodal causal attention mask, allowing Queries (visual features) to attend to each other, but not allowing attention to future Text tokens. Each Text token can attend to all Queries and previous Text tokens.

• The [CLS] token is replaced with a [DEC] token, serving as the first text token to indicate a decoding task.

• The output Query embeddings from the Image Transformer are fed into a binary classifier to obtain the corresponding logit.

• All logits are averaged to calculate the matching score.

• ITM uses a bidirectional attention mask, allowing all Query and Text tokens to attend to each other, so the output Query embeddings can represent the multimodal fusion information of image-text pairs.

Second-stage Pre-training: Generative Learning

1. Feature Projection: Use a fully connected layer to linearly project the Query output embeddings by Q-Former to the same dimension as the LLM's text embedding.

2. Input Construction: Prepend the projected Query embedding to the input text embedding.

3. Task Objective: Since Q-Former has been pre-trained and can extract visual representations most relative to language information, it acts as an information bottleneck, transmitting the most useful information to the LLM while filtering out irrelevant visual information, thus reducing the burden on the LLM to learn visual-language alignment.

• Decoder-based LLM: Used for pre-training with language modeling loss. The frozen LLM generates text based on Q-Former's visual representation.

• Encoder-decoder-based LLM: Used for pre-training with prefix language modeling loss. The text is divided into prefix and suffix parts. The prefix text and visual representation are concatenated as input to the LLM's encoder, and the suffix text serves as the generation target for the LLM's decoder.

Official Code