Scaling Law: Scaling laws in language models refer to the empirical observation that model performance improves predictably and smoothly as the model size, training data, and computational resources increase.
Traditionally, reward models have not been directly associated with scaling laws. However, this study explores a novel approach to improving reward models by establishing a connection between reward modeling and inference-time scaling laws. In other words, it investigates whether allocating more inference time to the reward process can lead to better overall performance.
Introduction
✨
The key challenge of applying reinforcement learning to LLM lies in obtaining accurate reward signals across various domains, not limited to verifiable problems or manually specified rules. This study improves reward quality by increasing inference-time computation and tries to find a way for inference-time scalability of the reward model (RM). This paper adopts a point-wise generative reward modeling (GRM) approach, and proposes a Self-Principled Critique Tuning (SPCT) learning method, leading to the development of the DeepSeek-GRM model. It also introduces Meta RM modeling to guide the voting process. Experimental results show that it outperforms existing methods and models in various reward modeling benchmark evaluations, without significant bias.
❓
Problem Background
Reinforcement learning is widely applied in LLM training, and reward modeling is a key component. However, it is difficult to obtain high-quality reward signals in general domains. Current reward modeling methods face limitations in input flexibility and inference-time scalability and have not fully explored the scalability of inference-time for reward models. Studies such as DeepSeek-R1 have also shown that whether during training or inference, as long as there is a high-quality, stable, and reliable reward mechanism, LLMs can achieve outstanding performance in specific domains.
The challenge of obtaining high-quality rewards: In specific domains, high-quality rewards mainly come from human-designed environments with clear conditions or precisely crafted rules for verifiable tasks (e.g., math and code tasks). In general domains, however, the standards of reward are more diverse and complex, often lacking clear reference benchmarks or factual grounding.
Challenges of general reward modeling:
Flexibility for different input types
Accurate reward generation across various domains
As inference-time computation increases, the reward signals can get better quality
Learning scalable behaviors for better performance-compute scaling.
Reward Model Classification and Pointwise GRM
In this study, the Reward Model (RM) is classified into two main categories:
A single Scoring Pattern is chosen to determine how the prompt data is input into the reward model, while one Reward Generation Paradigm defines how the model generates the output score. So, for the RM, there are a total of 6 possible combinations of these components.
⭐
Reward Generation Paradigms
Scalar: For a given Query + Response, a scalar reward value is assigned.
Semi-Scalar: For a given Query + Response, the reward model outputs a Critique (i.e., the reason for the score), and also generates a scalar reward value.
Refer to the Generative Verifiers paper: It uses prompt such as Is the answer correct? to ask GenRM-CoT to generate YES/NO token (it can be trained to reasoning and generate CoT first, and give the YES/NO token later). probability as the reward score. Then, it extracts the probability of the ‘Yes’ token.
Generative: A Critique is generated, and the reward is extracted from it, often written in a structured format within the Critique. Compared with the Semi-Scalar method, the Generative method asks the model to generate the score directly after reasoning.
⭐
Scoring Patterns
Pointwise: Each response is scored individually with a score.
Pairwise: Two responses are compared by assigning a relative score or selecting the better one from the pair.
⭐
Pointwise GRM:
Why choose the Generative reward paradigm + Pointwise scoring paradigm?
Based on the generation and scoring paradigms described above, different combinations have different scalability during inference and flexibility in input:
For input: Pairwise Pattern only considers the relative preference between two responses, lacking the flexibility to handle single or multiple responses as input. Therefore, methods like PairRM and LLM-as-a-Judge lack input flexibility.
For inference: the Scalar Paradigm makes it difficult to generate diverse reward signals for the same response (since the generated reward is fixed). This makes it hard to leverage reasoning-stage expansion methods that involve sampling (e.g., sampling multiple outputs for voting, hoping to capture the diversity within the critique).
Based on these reasons, this study chooses the combination of aGenerative reward paradigm + Pointwise scoring paradigm and namedPointwise GRM.
This is a commonly used approach when using models like GPT to generate data, which writing detailed prompts to make the model output a piece of reasoning and a score, then extracting the score. However, this paper emphasizes SPCT and MetaRM methods, which will described later.
⭐
How to Compute Pointwise GRM.
To scale computation during the inference phase, this paper focuses on a sampling-based approach. Multiple rewards are generated for the same Query–Response pair, and then these rewards are aggregated. Therefore, whether reward modeling enables inference-time scaling largely depends on whether different rewards can be obtained from multiple samples. The formula used in this paper for Pointwise GRM is as follows:
Where:
is the query content,
is the response,
is the reward function,
is the reward,
is the critique, which includes the reasoning and the final reward score generated by the model,
is the individual score for
extracts the reward from the generated result.
In this paper, the scores are discrete, using a 10-point scale (from 1 to 10).
Improving Reward Quality with Principles
Generalist RM needs to generate high-quality rewards not only in specialized domains but also in general domains (where reward standards are more complex and diverse). For general domains, like role-playing or writing, the criteria for evaluation become more complex, and there typically isn’t a single, fixed “right” answer.
This study uses Principles to guide reward generation, instead of relying on rule-based methods. The formula for principle-guided reward generation is as follows, where represents the principles. The reward score is generated by
A “principle” is a high-level, natural-language guideline that tells the reward model what “good” behavior looks like.
⭐
Example of Using Principles
Pick a task and two candidate outputs
User request : Translate to French: 'Good morning!'
Candidates :
: Bonjour !
: Bonne matin !
Predefined and Self-generate principles
We prompt GPT-4 with something like: Here are two translations of 'Good morning!' into French. What 2 high-level criteria (principles) should we use to judge them?
GPT-4 replies with (for example):
: The translation must be grammatically correct.
: It must convey the same meaning.
We may also predefine a principle:
: It should be idiomatic French.
Score each candidate under each principle. We then feed into our reward model prompt: For each of these translations, rate 1–10 how well it satisfies , , . Example score:
candidate
(grammar)
(meaning)
(idiomatic)
sum
10
10
9
29
8
6
10
24
Those summation become the reward , which is the model’s predicted reward scores .
Filter the principles automatically. Suppose we have a small set of gold comparisons where humans say Bonjour! () is always better than Bonne matin!. We check each principle list by seeing whether its induced scores agree with the human ranking.
If all three principles lead to the correct ordering, we keep them.
If, say, sometimes ranks higher than on this example, we drop .
After filtering we might be left with . Re-computing the averages:
candidate
p₁
p₂
average (filtered)
10
10
10.0
8
6
7.0
Baseline: no principles at all. We also run a baseline where we don’t give any criteria. We simply ask the model: On a scale of 1–10, how good is this translation?. And it might reply:
→ 8
→ 5
📌
Result and Potential Test-Set Leakage
in Per-Example Principle Filtering
In their pipeline, the authors first prompt GPT-4 to invent several natural-language “principles” tailored to each example (i.e. each prompt + its candidate answers). They then automatically filter out any principle that, when used to score that same example, fails to reproduce the known human preference ranking. Finally, they show that these filtered, hindsight-selected principles yield much higher agreement with the labels.
The problem is that the filtering criterion itself uses the very human labels it later evaluates, so it essentially hand-picked only those principles that make the model “get it right.” That’s a test-set leakage.
To avoid this circularity, we’d need to generate and filter the principles on a separate dev set, then apply the same set of principles (or a reward model trained with them) to an unseen test set. Only then can we claim that the boost in reward quality reflects true generalization, rather than simply overfitting to the ground-truth labels.
SPCT - Training
Self-Principled Critique Tuning (SPCT) enables LLMs to learn to generate adaptive and high-quality principles, thereby effectively guiding the generation of critique content. SPCT training includes two phases:
Cold-start rejective fine-tuning phase
Rule-based online RL phase
This paper integrates principles into the generation layer. In other words, principles are treated as part of the reward-generation process. The GRM model autonomously generates principles and then produces critique content based on these principles to extract rewards. The formulation is as follows:
The principles contain two parts (see the prompt input later):
Pre-defined principle: instruction adherence, usefulness, level of detail, relevance,
Generated principle based on the input query and responses.
This allows the reward generation process to adaptively align with the input. As model scale increases, generative reward models may produce finer-grained rewards under more reasonable standards (or principles), which is crucial for scalability during inference.
Prompt input (Appendix G) for DeepSeek-GRM:
📌
Rejective Fine-Tuning (RFT) as Cold Start
This part uses a pre-trained GRM (in this study, the pre-trained GRM is initialized using Gemma-2-27B) to repeatedly sample from the reward modeling data, consisting of a query and its corresponding responses.
N samples are drawn (generating principles + critiques and then extracting rewards).
Rejection strategy 1: reject predictions where the generated reward is inconsistent with the ground truth.
For example, the ground truth says response1 is better than response2, but the generated reward scores response2 higher. This set of principle + critic is considered wrong, and will be rejected.
The mathematical condition is as follows, where is the ground truth reward for the response to query , assuming there is only one maximum ground truth value:
Rejection strategy 2: If all of the N samples correctly evaluate the query and responses (i.e., overly simplified), the data will be rejected.
Hinted Sampling for hard data: For queries and responses that are difficult to generate rewards for, this paper appends the ground truth to the prompt for hinted sampling, and samples only once. Hinted sampled trajectories sometimes shortcut the generated critique in reasoning tasks, which highlights the need and benefits of online RL for the GRM.
📌
Rule-based online RL
After the cold-start phase, the GRM proceeds to online GRPO training. It uses the original GRPO setup (See Section 3.4.1 GRPO), but removes the format reward, replacing it with a larger KL penalty coefficient to ensure format correctness and avoid severe bias. The rule-based reward is defined as follows:
For a given query and responses , the reward for the output is:
Where the scores are extracted from the output . This reward function is similar to the rejective strategy in RFT and is used to fine-tune the GRM's principle and critique generation toward better identifying the optimal response.
SPCT - Inference
This study further explores inference-time scaling based on sampling.
⭐
Voting with Generated Rewards
Scaling the Inference-Time Computing
A simple voting method is used, which samples multiple trajectories and then aggregates the extracted rewards by simply adding them together:
The voting process basically expands the reward space by a factor of , allowing the generative reward model to produce a large number of principles. This helps improve the quality and granularity of the final reward. One intuitive interpretation is that each principle can be seen as a different evaluation perspective. More principles may more accurately reflect the true distribution of quality, thereby achieving scalable effect.
⭐
Meta Reward Modeling Guided Voting
Remove the Bad Principles
Due to randomness and model limitations, multiple sampled principles and critiques may have bias or inconsistent quality. Therefore, this paper trains a pointwise scalar meta RM (The meta RM is also trained on Gemma-2-27B) to evaluate the correctness of the GRM-generated principles and critique content, outputting a Meta Reward value, selecting the Top , and then performing voting.
Meta RM is trained via binary classification using non-hinted samples from the RFT phase, and also sampled trajectories from the DeepSeek-GRM.
Overall, the derivation of DeepSeek-GRM-RFT, DeepSeek-GRM, and Meta RM in the SPCT pipeline is illustrated as:
Experiments
⭐
Performance on RM Benchmarks
Scalar and semi-scalar reward models show bias across different benchmark evaluations. For the verifiable task (PPE Correctness), their performance is better than all generative reward models, but on other benchmarks, they each perform poorly. SPCT improves the GRM's ability to generate rewards in general domains, and compared to scalar and semi-scalar reward models, its bias is significantly smaller.
Overall results of different methods and models on RM benchmarks. Underlined numbers indicate the best performance, bold numbers indicate the best performance among baseline and method proposed in this study, and italicized font denotes scalar or semi-scalar RMs. For meta RM guided voting (MetaRM), .
🔬
Inference-Time Scalability
With up to 8 sampling rounds, DeepSeek-GRM-27B achieves the largest performance improvement over greedy decoding. Further increasing to 32 sampling shows strong potential for performance gains. MetaRM also proves effective in filtering out low-quality trajectories.
Inference-time scalability results of different methods on RM benchmarks.
Inference-time scaling performance with different RMs on all tested RM benchmarks.
From the results shown above, we observe the following:
The Scalar Paradigm does not support sampling multiple outputs for voting, limiting its ability to capture diverse feedback.
The Semi-Scalar + Pointwise scoring paradigm (represented by the green curve) shows no significant improvement as the number of samples increases, indicating limited benefit from sampling.
GRM demonstrates the best inference-time scalability. The performance improves consistently as more samples are generated and evaluated.
🔬
Ablation Study
The ablation results for different components of SPCT are shown in the figure on the right
Even without cold start RFT, GRMs fine-tuned with general-purpose principles still show significant performance gains after online RL (from 66.1 to 68.7).
Non-hinted sampling is more important than Hinted Sampling — possibly because hinted sampling often leads to overly simplified trajectories.
Principle generation plays a key role in both greedy decoding and inference-time scaling for DeepSeek-GRM-27B.
Meta RM Voting performs robustly under different values of K.
Ablation studies for different components of the proposed SPCT. Bold numbers indicate the best performance.
🔬
Scaling Inference and Training Costs
Using 32-sample voting with DeepSeek-GRM-27B achieves performance comparable to a 671B MoE model.
MetaRM Voting achieves optimal results with just 8 samples, showing that DeepSeek-GRM-27B is effective for inference-time scalability compared to scaling up model size.
This paper also tested 300-sample test sets on DeepSeek-R1 and found its performance was even worse than a 236B MoE RFT model, indicating that longer reasoning chains during inference cannot significantly improve general-purpose reward modeling performance.
Inference-time scaling performance v.s. training-time scaling performance on the Reward Bench benchmark.
Left: Inference-time scaling results of DeepSeek-GRM-16B and DeepSeekGRM-27B ().
Right: Training-time scaling results with different model sizes, using greedy decoding except DeepSeek-R1.
Conclusion
This paper introduces SPCT to enhance the inference-time scalability of general-purpose reward modeling. With the help of Rule-based Online RL, SPCT enables the GRM to autonomously generate adaptive principles and critiques, significantly improving the quality of rewards across domains and the scalability during inference. Future research directions may include integrating GRMs as general-purpose reward interfaces into online RL pipelines, exploring policy models for inference-time cooperative scaling, or using GRMs as reliable offline evaluation tools for foundation models.
AI Roadmap
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP
OpenAI ChatGPT Progression
