Lazy loaded image10.2 VLM-R1 Reproduction Results

Hardware Information

Component
Specification
CPU
Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz
GPU
2 × A6000 (48GB)

Python Library

Task: Referring Expression Comprehension (REC)

Referring expression comprehension involves identifying and locating the target object in an image based on a natural language description. For example, given an input like "the student wearing yellow standing next to the car", the output would be the coordinates of that student in the image.

Script

Difference

  • GPU number
    • Official: 8
    • Ours: 2
  • per_device_train_batch_size
    • Official: 8
    • Ours: 4
  • gradient_accumulation_steps
    • Official: 2
    • Ours: 4
  • Total Batch Size
    • Official: 128
    • Ours: 32

Training Curve in Wandb Logs

notion image
This quite reproduces the official training curve, as shown below.
notion image
You can observe that the loss starts from 0 and gradually converges to 0.01, which is explained in 10.1 Why GRPO training loss starts at 0.

Evaluation (GRPO)

GRPO Performance comparison. Note that the 2500 step of checkpoint is not saved so we do not report it, which might be because of hard disk writing issue.
GRPO Performance comparison. Note that the 2500 step of checkpoint is not saved so we do not report it, which might be because of hard disk writing issue.
The orange lines are the official reported performance, and the blue ones are our reproduction. You can observe with lower batch size, the convergence is slower, but eventually, the model still reaches an even better level of performance on both in-domain and out-of-domain test data.

Evaluation (SFT)

SFT Performance comparison.
SFT Performance comparison.
Still, the orange lines are the official reported accuracies. Our reproduction of SFT shows that the performance of the SFT model on in-domain data will keep increasing and can even compete with the GRPO-trained model. But as the official reported, the performance on out-of-domain data is indeed worse than that of GRPO. This might suggest that the RL (e.g., GRPO) training method has a better performance trade-off between the in-domain and out-of-domain data.
Prev
10.1 Why GRPO training loss starts at 0
Next
10.3 KL(P||Q) or KL(Q||P)?
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant