Lazy loaded image11.4 EVA

Paper: EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
notion image
Objective: Select a Masked Image Modeling (MIM) visual pre-training task with excellent transfer performance to downstream tasks.
Candidate Methods:
  • Recover masked semantic visual token features: Based on CLIP visual features, recover the masked semantic features.
  • Feature Distillation: Perform feature distillation from strong pre-training representations.
Experimental Results:
  • Additional tokenization (i.e., BEiT) on CLIP features is not necessary for performance improvement in downstream tasks.
  • Longer feature distillation time during pre-training does not provide consistent performance improvement.
  • Optimal choice: Directly reconstructing the masked CLIP visual features based on visible patches.
Note that this MIM task is not proposed for the first time, it has been studied in MVP and MILAN. EVA demonstrated that this task can be extended to billions-scale parameters and millions-scale images, and does not require:
  • Semantic quantization/tokenization;
  • Explicit use of image-text pairs as pre-training data.

Model Architecture

  • Model Framework: EVA is a common ViT with 10 billion parameters.
  • Features:
    • Complies with ViT giant and BEIT-3 visual encoders.
    • Relative position embedding and layer-scale are not used during pre-training.

Pre-training Objective

  • Task: Reconstruct masked CLIP visual features based on visible image patches.
  • Masking Strategy:
    • Use [MASK] tokens to corrupt input patches.
    • Employ block-wise masking, with a masking ratio of 40%.
  • Target Features: From OpenAI CLIP-L/14 vision tower (originally trained on 224x224 images).
  • Output Processing:
    • First, normalize EVA's output features.
    • Then, project them to the same dimension as CLIP features via a linear layer.
  • Loss Function: Negative cosine similarity.

Data

  • Datasets:
    • CC12M and CC3M: Image data only, no captions.
    • COCO and ADE20K: Training set data only.
    • ImageNet-21K and Object365: Image data used.
  • Total: 29.6 million images.
  • CLIP Features: Trained on 400 million pairs in a self-supervised behavior. EVA implicitly utilizes the image-text pairs from this dataset.

Hyperparameters

  • Optimizer: Adam, weight decay 0.05.
  • Learning Rate: Peak value 1e-3, decayed according to a cosine learning rate schedule.
  • Data Augmentation: RandResizeCrop (0.2, 1).
  • Color Jitter: Not used.
Prev
11.3 BLIP
Next
11.6 BLIP-2
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant