🔗9.3.7 o1-Coder

Paper link: o1-Coder: An o1 Replication for Coding
Code link: https://github.com/ADaM-BJTU/O1-CODER
⭐
A team from Beijing Jiaotong University introduced o1-Coder, a project aiming to duplicate OpenAI’s o1 model and focus on code tasks. o1-Coder integrates RL and MCTS to enhance LLMs’ System-2 reasoning ability. The framework of o1-Coder mainly includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and finally fine-tuning the policy model. Currently, part of the code is open-source, with plans to release updates, including RFT training code (stopped updating until December 10, 2024).

Beckground

  • Impact of the GPT-o1 Model: OpenAI’s o1 model demonstrates remarkable System-2 reasoning ability (deeper, deliberate, and logical thinking steps). It has advanced the development of AI reasoning model research, attracting numerous duplication attempts from academia and industry. The work of o1-Coder builds upon this reason.
  • Limitations of Existing Models and Solutions: Traditional LLMs mostly exhibit System-1 capabilities (fast, intuitive, but less accurate thinking patterns) and lack reasoning steps. Conventional methods like Chain of Thought (CoT) prompting and Supervised Fine-Tuning (SFT) have limitations. Therefore, this paper explores RL as a solution to generate and optimize reasoning data, especially for coding tasks.
notion image

Method

✨

Framework Overview

This study integrates RL (Reinforcement Learning) and MCTS to achieve self-play, enabling the model to continuously generate reasoning data and enhance its System-2 capabilities. However, applying self-play to code generation requires addressing two key challenges: 1) How to evaluate results, 2) How to define reasoning step actions (thinking or search behaviors)
  • Generally, evaluating code generation results is not as straightforward as evaluating math (with clear answers) or games like Go (with explicit rules). It requires a corresponding testing environment to run the generated code and check correctness through enough test cases. Therefore, this paper proposes training a Test Case Generator (TCG).
  • The design of the reasoning step actions has two paradigms:
      1. Think before Acting: the model first organizes a complete chain of thought (CoT), then generates the final answer.
      1. Think while Acting: the model generates partial answers while reasoning.
      This paper chooses the first approach.
The overall framework includes generating reasoning data iteratively, training PRM (Process Reward Model), and updating strategies.
Self-Play+RL training framework
Self-Play+RL training framework
✨

Test Case Generator

The training of TCG mainly consists of two stages: SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization, see Section 3.5 DPO). It uses DeepSeek-1.3B-Instruct as the base model and adopts parameter-efficient fine-tuning (QLoRA). With rank=1, QLoRA is applied to the following matrices: q_proj, o_proj, k_proj, v_proj, gate_proj, up_proj, down_proj.
  • SFT Stage: The main goal is to ensure that TCG’s output follows the predefined format, so it can accurately parse and extract generated test cases. The training data comes from the TACO dataset, and prompt templates are reorganized as shown below.
      • notion image
  • DPO Stage: The main goal is to guide the model to generate test cases that meet specific preferences, thereby improving the performance and reliability of the test case generator. The training data uses a pre-constructed preference dataset, with the following rules:
    • chosen = directly sampled test cases where the input and output (three samples) completely match.
    • rejected = test cases where the input and output from the three samples are shuffled (since a test case typically includes corresponding input and output), resulting in incomplete, mismatched test cases.
✨

Reasoning-enhanced Code Data Synthesis

  • Reasoning process based on pseudocode: Pseudocode can serve as an intermediate representation between natural language descriptions and actual code, providing a more abstract and concise way to express algorithms or program logic. Therefore, this paper designs three reasoning step actions based on pseudocode:
    • Action 1: Define algorithm structure using pseudocode
    • Action 2: Modify pseudocode
    • Action 3: Generate code from pseudocode
      • These actions ensure that the model uses pseudocode as a cognitive tool during reasoning, strengthening its ability to handle complex code generation tasks. The model may need to repeatedly select Action 2 throughout the reasoning process to iteratively refine and adjust the pseudocode until it becomes usable for the final code generation step. A specific action selection prompt template is shown below.
      notion image
      Example:
      notion image
       
      notion image
  • Reasoning Process Data Synthesis: Mainly uses the MCTS algorithm to generate reasoning process data with intermediate rewards (values).
    • : evaluation result of the reasoning path up to step .
    • : executable code derived from the final step .
      • This paper adopts the standard MCTS rollout algorithm. For a problem , when the final step is reached, a complete reasoning path is expressed as . The value of the final state is calculated based on two main criteria:
    • Compilation success rate: 1 if compilation succeeds, otherwise 0.
    • Test case pass rate: fraction of test cases successfully passed.
      • Finally, is calculated by:
      Once the final state value is obtained, backpropagation is performed along the reasoning path to get the value , and at each node are normalized to the range [0,1].
  • Policy Model Initialization: During the construction of the reasoning process data, a batch of cases with can be obtained, which means the final results successfully compile and pass all test cases.
      • This batch of data is then used to perform SFT training on the base model to initialize the policy model, which will later undergo iterative updates.
 
✨

PRM (Process Reward Model) Training

You can check Section 3.4.1 GRPO - Advantage Function Estimation in GRPO for more about PRM.
PRM can be trained using two methods:
  • Point-wise: The collected data is organized as:
      • Where is the number of samples. PRM is trained with the cross-entropy loss of the label and the reward model’s output :
      Here, is the normalized prediction score output by PRM.
  • Pair-wise: For a node at depth in the MCTS search tree, the preference pair data is organized as:
      • Here, and represent reasoning steps with relatively higher and lower value estimates obtained during the tree search. PRM is trained with the following objective:
      Here, the score function is not normalized.
✨

RL-based Policy Model Improvement

This paper models the code generation task as an MDP (Markov Decision Process), formally represented as:
  • : vocabulary, denotes a single token generated by the model.
  • : action space.
  • : state space (a set of token sequences). States are token sequences.
  • : the problem (initial state).
  • : an action (a reasoning step, including action type and its CoT).
The state transition function:
The specific operation is: concatenate the current action to the current state to form the next state .
The reward function evaluates the quality of intermediate reasoning steps.:
The signal function combines PRM (Process Reward Model) and ORM (Outcome Reward Model) to generate the final reward signal:
Here, is the outcome reward, determined by the test pass rate of the generated code. is a coefficient that balances outcome rewards and process rewards with respect to reasoning steps. This coefficient may decrease over time. As the model improves its solutions, more weight is gradually assigned to process rewards. At the same time, as the model approaches the optimal policy, reliance on outcome rewards is reduced. Subsequently, PPO or simplified DPO can be used for policy improvement.
The paper defines a dedicated RL environment for code generation tasks. Actions are jointly driven by process-based rewards (encouraging intermediate reasoning steps) and outcome-based rewards (reflecting the correctness of final code). This dual reward structure helps the model improve its code generation ability over time.
Finally, the updated model can be applied with MCTS to generate new reasoning data, which is added back to the dataset, further updating PRM. Thereby forms a self-play training cycle that continuously enhances reasoning capabilities.

Experiment

🔬
This Paper Only Provides One Experimental Result
Using an MCTS-based sampling strategy, the paper compares standard CoT reasoning with pseudocode-enhanced CoT on the MBPP benchmark, evaluating two metrics Pass@1 and ASPR (Average Sampling Pass Rate), i.e., the average pass rate of the final step in the reasoning path. The experimental results show a significant improvement in ASPR, indicating that pseudocode strengthens the overall reasoning process, especially in pathways that modify pseudocode into correct final outputs.
However, the Pass@1 metric dropped substantially, suggesting that the overall effect was not very good.
Pseudocode-based code generation results on the MBPP Benchmark.
Pseudocode-based code generation results on the MBPP Benchmark.

Discussion

📌

Lessons Learned

  • Key Role of Data: AI development focuses on improving the efficiency of computation-intelligence conversion. Data acquisition becomes critical, enabling the collection of missing data or the exploration of new data types.
  • Beyond Human Data: The o1 model demonstrates that through RL exploration, it can learn the reasoning processes behind data, surpass the limitations of human language data, and enhance model capabilities.
notion image
📌

Opportunities and Challenges

  • Opportunities: The self-play RL framework can help extend System-2 reasoning to multi-domain tasks, improving model safety and enabling exploration of System-2 alignment.
  • Challenges: In practical applications, the o1-type models face difficulties due to reward function design and environment state updates. This requires building world models to predict states. However, constructing such models remains highly challenging, though progress in related fields offers hope.
 
 
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant