🔗9.3.2 Let’s Verify Step by Step

✨
PRM Process Supervision Reward Model: Let’s Verify Step by Step
Researchers from OpenAI explored the effectiveness of process supervision and outcome supervision when training language models to solve complex multi-step reasoning problems. They found that process supervision during model training is significantly better than outcome supervision in solving challenging problems in the MATH dataset.
Paper link: https://arxiv.org/pdf/2305.20050

Research Background:

❓
Background:
  • Logical errors and illusions in large models during step-by-step reasoning:
      • Large models can solve complex tasks through step-by-step reasoning or thinking (Chain of Thought, CoT), but even the most advanced models can still generate logical errors and illusions, which requires using supervised rewards to distinguish between reasonable and unreasonable outputs.
  • Lack of detailed comparative studies on process supervision vs. outcome supervision in reward models:
      • Training reward models mainly uses outcome supervision and process supervision. Process supervision offers more accurate feedback, is easier to interpret, and better aligns with human thought chains, but previous research found the two yield similar final performance in math domains.
Research Objective: To compare the effectiveness of outcome supervision and process supervision in training models to solve math problems, and to study the impact of active learning on process supervision.

Method

💡

Recall of Reward Model Construction: ORM and PRM

Note: This paper does not explicitly provide the ORM and PRM models. The formulation below comes from the paper: Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Also see Section 3.4.1 GRPO, the overall idea for this section is the same, but the details are not.
To evaluate the quality of large model outputs, there are generally two types of evaluators: Outcome-supervised Reward Models (ORMs) and Process-supervised Reward Models (PRMs). ORM evaluates the entire solution, while PRM evaluates each reasoning step in the process.
  • ORMs: Directly determine whether the final result of a solution is correct or incorrect (even if there are reasoning errors in the middle, the final result might be correct). From the generator, a fixed number of samples are drawn as a solution. ORM is trained to evaluate whether each solution is correct by checking the final answer for correctness.
      • ORM Objective Function: Given a math problem and a solution , ORM assigns a scalar score to indicate whether is correct. ORM typically uses cross-entropy loss for training (instead, if the reward is not a rule-based model, but a model, the reward is not a binary value):
      Where:
    • is the label ( if is correct, otherwise )
    • is the probability output by ORM (via a sigmoid function)
  • PRMs train the model to evaluate each reasoning step, enabling training directly within autoregressive models. When reaching the end-of-sequence token, PRMs are trained to predict the correctness of each individual step. During testing, a forward pass of the entire solution through PRM provides a step-level correctness evaluation. The PRM score for the solution is then calculated as the product of the probabilities that each step is correct.
      • PRM Objective Function: More formally, PRM assigns a score to each reasoning step, and is typically trained using the following formula:
      Where:
    • is the label for the step of the solution ( if correct, if incorrect)
    • is the score (sigmoid output or probability) assigned by PRM to the step
    • is the total number of reasoning steps
  • The problem of unequal supervision signals between ORM and PRM: When providing process supervision, this study supervises only up to the first incorrect step, making the comparison between outcome supervision and process supervision more straightforward.
    • Why is it only labeled up to the first mistake?
      • If labeled every mistake in a wrong solution, PRM would get much more information than ORM, making the comparison unfair.
      • By stopping at the first incorrect step, it ensures both methods see the same amount of error information.
    • How this work
      • Correct solutions:
        • ORM sees correct.
        • PRM sees step 1 correct, step 2 correct, …, step K correct.
        • Both effectively get full credit for every step.
    • Incorrect solutions:
      • ORM sees wrong.
      • PRM sees step 1 correct, …, step j wrong (and then stops labeling).
      • Both know there is an error, but PRM also pinpoints where it first went off track.
    • Why stop at the first error?
      • Letting PRM label beyond step would give it strictly more detail than ORM.
      • In practice, asking a human to find the first mistake is about the same effort as simply judging “right vs. wrong” at the end. So this choice also keeps the annotation cost reasonable.
💡

Method:

  • Model Scale: Experiments are conducted under both large-scale and small-scale model settings. The large-scale model finetuned from the GPT-4 model, used to obtain the most reliable ORM and PRM. The small-scale model has a similar design but 200× less computational cost, and both are further fine-tuned on the MathMix dataset.
  • Generator: A generator is trained to produce step-by-step solutions in a newline-delimited step-by-step format. For MATH training problems, a small set of solutions is generated, and correct answers are selected to fine-tune the base model.
  • Evaluation: N solutions are sampled, and the final score is based on whether the correct answer was chosen.
  • Data Collection
    • Process Supervision Data: Human annotators labeled each step of the MATH problem solutions generated by a large-scale generator as correct, incorrect, or neutral. The resulting dataset is referred to as PRM800K.
    • Selection Strategy, Active Learning: To maximize the value of human feedback, seemingly reasonable but actually incorrect solutions were selected for annotation. PRM was retrained iteratively during data collection.
      • Baseline Strategy: Every reasoning step is labeled with human feedback.
      • Optimized Strategy, mining high-value negative samples: When labeling, prioritize the data that might mislead the reward model. Solutions that are obviously wrong yield less informative feedback.
      • Iterative Reward Model Training: In each iteration, generate solutions per problem, but only show annotators the top most plausible incorrect ones. This paper tests top-K filtering at the:
        • problem level (each problem has candidate solutions)
        • global level (top total across all problems, with uneven distribution across problems)
      A screenshot of the interface used to collect feedback for each step in a solution.
      A screenshot of the interface used to collect feedback for each step in a solution.
      Two solutions to the same problem, graded by the PRM. The solution on the left is correct while the solution on the right is incorrect. A green background indicates a high PRM score, and a red background indicates a low score. The PRM correctly identifies the mistake in the incorrect solution.
      Two solutions to the same problem, graded by the PRM. The solution on the left is correct while the solution on the right is incorrect. A green background indicates a high PRM score, and a red background indicates a low score. The PRM correctly identifies the mistake in the incorrect solution.

Experiments

🔬

Large-Scale Supervised Learning

How the two models were trained
  • PRM: trained once on PRM800K using step-level (process) labels.
  • ORM: trained once on a big outcome-supervision set built by taking 100 uniformly sampled solutions per training problem from the generator and labeling them by final correctness. This set does not overlap with PRM800K and is an order of magnitude larger.
    • They note that training ORM purely on PRM800K solutions would be biased (that dataset over-represents wrong steps due to active learning).
    • They tried mixing PRM800K solutions into the ORM set. It didn’t help ORM performance.
The result below shows that:
notion image
  • x-axis (N): how many candidate solutions are sampled per problem at inference.
  • PRM curve: for each problem, sample N solutions, score each step-by-step with the trained PRM, pick the highest-scoring solution, and check if it’s correct. Plot accuracy vs N.
  • ORM curve: do the same, but score each whole solution with the trained ORM (a single final reward).
  • Majority voting baseline: ignore reward models, and just take the most common final answer among the N samples and see if it’s correct.
As shown in the figure, where N represents the number of sampled solutions per question, as N increases, the performance gap between PRM and ORM becomes larger. This indicates that, compared with ORM and majority voting, PRM is more efficient at finding the correct answer among many candidate answers.
🔬

Small-scale Synthetic Supervised Learning

Fair Comparison Between PRM and ORM
Because of differences in dataset construction, supervision methods, and evaluation methods, a direct comparison between ORM and PRM is not straightforward.
Reasons for the Problem:
  • The training sets for ORM and PRM cannot be directly compared. PRM’s training set is constructed through active learning, which biases it toward incorrect solutions and is also one order of magnitude smaller.
  • ORM evaluates the final answer’s correctness, but errors in the reasoning process of the problem may lead to unfair judgments.
  • The cost of collecting human feedback is high, making it difficult to eliminate these factors via annotation.
Solution: Use a large-scale PRM ( ) to supervise smaller models for an ablation study. Four models or methods are used:
  • PRM + Active Learning
    • Start with a very small , train it on a tiny labeled dataset (from human labels or ).
    • Generate candidates: For each math problem, sample ~1000 candidate solutions (step-by-step reasoning traces) from the generator model.
    • Select informative examples (active learning step)
      • Use to score all 1000 candidates.
      • Keep 80% that look “convincing but wrong” (tricky errors), and 20% of the “most convincing overall” solutions.
      • Idea: focus on hard cases.
    • Label with : Use a strong “teacher” PRM (, trained on 800k human-labeled steps) to give step-level labels. These labeled traces become the training set.
    • Train the on those step-level labels
  • PRM ( supervised)
    • Uniformly sample solutions per problem.
    • Label with (step-level process supervision), and use the label to train .
  • ORM ( supervised)
    • Uniformly sample solutions per problem.
    • Label with , but only use final-answer correctness (ignore step-level info), and use the final label to train .
      • gets richer supervision signals than .
  • ORM (final-answer supervised)
    • Uniformly sample solutions per problem.
    • Label with ground-truth final answer check (true/false only). No model is trained.
Experimental Results and Analysis:
Left figure: Vary the training data size () while keeping test-time candidates fixed, shows how more labeled data improves PRM vs ORM. Right figure: Fix the training data size (200 samples/problem) and vary the test-time candidates (), shows how PRM vs ORM scale when given more solutions to choose from.
Left figure: Vary the training data size () while keeping test-time candidates fixed, shows how more labeled data improves PRM vs ORM. Right figure: Fix the training data size (200 samples/problem) and vary the test-time candidates (), shows how PRM vs ORM scale when given more solutions to choose from.
  • A direct comparison of outcome supervision vs. process supervision shows that process supervision consistently outperforms both forms of outcome supervision across all dataset scales. In the figure right, using for outcome supervision makes final-answer checking more effective: provides better supervision by ensuring the correctness of results and penalizing reasoning errors.
  • Considering the impact of active learning, active learning improves the data efficiency of process supervision by about 2.6x. However, the limited diversity of training samples may constrain its effectiveness. Iterative retraining of the selector model indicates potential for further improvement, though more research is needed. The authors see this as a promising direction.
  • Final-answer supervision is noisy because many wrong-reasoning solutions still reach the correct answer, while supervision avoids these false positives, giving cleaner labels. So ORM ( supervised) outperforms ORM (final-answer supervised).
🔬

OOD Generalization

Evaluating large-scale ORM and PRM on 224 Out-of-Distribution STEM problems
  • Method: A held-out set of 224 STEM problems was used to evaluate large-scale ORM and PRM. These problems came from recent AP Physics, AP Calculus, AP Chemistry, and AMC10/AMC12 exams. Since these tests were released after the training dataset was collected, it is confident that the models had never seen them before.
  • Conclusion: PRM shows better generalization ability than ORM and majority voting. This demonstrates that PRM can tolerate moderate distribution shifts, maintaining strong performance on new test problems.
Out-of-distribution generalization is measured using recent STEM tests. The outcome-supervised RM, the process-supervised RM, and majority voting using 100 test samples per problem are evaluated.
Out-of-distribution generalization is measured using recent STEM tests. The outcome-supervised RM, the process-supervised RM, and majority voting using 100 test samples per problem are evaluated.

Discussion

⭐

Advantages of Process Supervision

  • Clear credit assignment: Process supervision provides richer signals, explicitly pointing out which steps are correct and which are wrong, making credit assignment easier and helping models generalize better.
  • Impact on AI alignment: Process supervision is more likely to produce interpretable reasoning, since it encourages models to follow a process endorsed by humans. It is also inherently safer, directly rewarding aligned chains of thought rather than relying on outcomes as a proxy. In contrast, outcome supervision is harder to scrutinize and less precise, and can even risk misalignment if models learn to exploit imperfect reward signals. Unlike many alignment methods that reduce performance (the “alignment tax”), these results show that process supervision has a negative alignment tax: it improves performance while also enhancing alignment, which could promote broader adoption.
⭐

Contributions of This Work

  • Demonstrates that process supervision can train more reliable reward models: the process reward model solved 78% of representative subsets in the MATH benchmark.
  • Shows that large reward models can reliably approximate small human-supervised reward models, and can be used for large-scale data collection and distillation research.
  • Highlights that active learning significantly improves the efficiency of process supervision.
  • Released the PRM800K dataset to support further research.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant