🔗9.3.1 STaR: Self-Taught Reasoner

STaR: Self-Taught Reasoner, Bootstrapping Reasoning With Reasoning
Date Published: March 28, 2022
✨
Step-by-step "Chain-of-Thought" reasoning processes can improve language models' performance on complex reasoning tasks such as math or commonsense QA. However, existing prompting methods either require building large reasoning datasets or sacrificing accuracy by using only few-shot (few-example) prompting.
Researchers from Stanford and Google proposed the Self-Taught Reasoner (STaR) approach, which improves model performance on complex reasoning tasks by leveraging a small number of reasoning examples and a large dataset without reasoning processes, enabling the model to learn and improve from its own generated reasoning.

Introduction

❓
  • The Importance of Reasoning for Model Performance:
      • Human decision-making is often the result of extended chains of thought: breaking a problem into subproblems and reasoning through them step by step to arrive at a conclusion. This process is known as chain-of-thoughts (CoT). Recent research shows that generating clear intermediate reasoning steps (rationales) (example figure below) before producing a final answer can significantly enhance the performance of language models on complex reasoning tasks (e.g., math, common-sense QA).
      Few-shot examples promot method
      Few-shot examples promot method
  • Problems with Existing Rationale Generation Methods:
    • Building Fine-tuning Datasets: One approach is to assemble a dataset of rationale-annotated examples and fine-tune a model to generate rationales. However, manually creating such a dataset is expensive, and automatic template-based methods, which rely on known general solutions or hard-coded prompt formats, have limited generalizability.
    • In-context Learning: Using handwritten or automatically generated templates to create a set of rationale generation examples as prompt templates, the LLM follows these examples for internal rationale reasoning to arrive at the final answer. Although including a few reasoning exemplars in the prompt can improve accuracy, models still lag behind those fine-tuned on large-scale rationale datasets.

Method

The method mainly consists of two parts:
⭐

Rationale Generation Bootstrapping

Method:
  • Starting from a small prompt set that contains a few examples with intermediate rationales , and using a pre-trained LLM and an initial dataset of question-answer pairs (note ).
  • Using a few-shot prompt style, each example in the prompt set is concatenated with the question , and then the LLM is prompted to generate an intermediate rationale and a final answer .
  • If , then the generated rationale is considered higher quality, and only those that lead to the correct answer are retained as rationales.
  • The selected data is then used to fine-tune model , allowing the updated model to continue generating better rationales in the next iteration. This process is repeated iteratively.
Note: New data should only be collected after the initial is fine-tuned to avoid overfitting.
STaR can be seen as an approximation to an RL-style policy gradient objective:
By filtering out the gradients from incorrect answers, it reduces estimator variance and applies multiple gradient updates, making the method simple and broadly applicable. can be viewed as a discrete latent variable model:
The LLM first samples a latent rationale before predicting . Then the indicator reward function is given, i.e., the reward equal to 1 only if the sampled answer matches the ground truth . The reward implicitly penalizes rationales that fail to produce the correct answer.
Expected total reward over the dataset:
The gradient of the objective:
The part in blue corresponding to rationalization.
The part in blue corresponding to rationalization.
⭐

Rationalization:

  • Problem: The model only trains on examples where it can produce the correct answer. When it encounters new questions in the training set that it cannot solve, learning will stop. The algorithm cannot extract any training signal from failed examples.
  • Solution: To address this, when the model fails to correctly answer a question, it is instead provided with the correct answer as a prompt, prompting the model to generate a rationale in reverse (rationalization).
      • These generated rationales are added to the dataset and then combined with previously selected data for fine-tuning the model again. This helps the model better solve new problems and enhances the quality and scale of the dataset.
A few-shot prompt hint for rationalization, using the rationale from, with its hint included in green, followed by the rationale and the answer generated by the model.
A few-shot prompt hint for rationalization, using the rationale from, with its hint included in green, followed by the rationale and the answer generated by the model.

Experiment

Experimental Setup

  • Model and Datasets: GPT-J (a 6B-parameter model) is used as the base language model, and experiments are conducted on the following datasets:
    • Arithmetic: problems generated by the author
    • CommonsenseQA: commonsense reasoning
    • GSM8K: grade school math
  • Experimental Protocol: For different tasks, data generation, sampling, rationale generation, and rationalization operations are performed according to the specified rules. The model’s performance is then recorded.
🔬

1. Symbolic Reasoning Results on the Arithmetic Dataset

The Arithmetic task involves calculating the sum of two n-digit integers, with examples shown in the figure. The paper uses few-shot prompts with numbers from 1 to 5 digits. During Rationalization, the correct answer is given as the target, and the LLM is asked to generate a detailed step-by-step reasoning process from scratch.
After 16 iterations post-training, STaR achieved an overall accuracy of 89.5% on the Arithmetic dataset. For reference, a baseline trained on 10,000 examples without rationales only reached 76.3% accuracy. In contrast, pre-training few-shot prompting methods (with rationales) have less than 1% accuracy on two-digit addition, with performance dropping toward zero as the number of digits increased.
A visualization of a 3-digit arithmetic problem with a scratchpad.
A visualization of a 3-digit arithmetic problem with a scratchpad.
Rationalization enables the model to rapidly improve its accuracy and allows it to learn a variety of long-form problems, achieving good performance even on out-of-distribution problems.
A visualization of the accuracy of n-digit summation with each iteration of STaR with and without rationalization for arithmetic.
A visualization of the accuracy of n-digit summation with each iteration of STaR with and without rationalization for arithmetic.
🔬

2. Natural Language Reasoning: Results on the CommonsenseQA Task

CommonsenseQA (CQA) includes a variety of questions that require reasoning based on standard world knowledge. Human accuracy on this task is around 89%.
STaR performs well on this task, outperforming the GPT-J baseline that directly predicts answers, and approaching the performance of the much larger GPT-3 model (about 30 times larger).
Rationalization further improves performance, which makes the model’s generated reasoning becomes more coherent. Human evaluations show that STaR improves reasoning quality. However, certain failure patterns exist, most notably”
  • Vague or non-arguments, such as when the model claims the question itself implies the answer. For example:
    • Question: Is the sky blue?
    • Model Output: You wouldn’t ask that unless the sky is blue.
  • Circular reasoning
  • Overly specific examples in place of general principles.
notion image
🔬

3. Mathematical Reasoning: Results on the GSM8K Task

GSM8K includes 7,473 training examples and 1,319 test examples of grade-school-level word problems. These math problems are given in natural language and typically require 2 to 8 steps to reach the final answer, combining arithmetic and commonsense reasoning skills.
STaR significantly improved performance on the GSM8K dataset, outperforming both few-shot reasoning and baseline models that directly predict answers.
The step-by-step reasoning generated by the model typically matches human problem-solving steps (53–57% agreement across iterations), and sometimes the model uses fewer steps, often by skipping steps or occasionally finding alternative solutions (figure right). However, rationalization operation does not yield a significant performance improvement on this task.
notion image

Discussion, Challenges, and Conclusion

💡

Discussion and Challenges

  • Effect of Rationalization: Rationalization helps the model to reverse-engineer a solution. From a mathematical perspective, it allows the model to explore better reasoning paths. However, further research is needed to understand its connection with RL objectives.
  • Effect of Sampling Temperature: Increasing the sampling temperature can expand the training data, but may cause the model to produce incorrect reasoning and reduce generalization ability, especially in structured tasks. It also results in low computational efficiency.
  • Effect of Few-shot Prompts: Including few-shot prompts in sampling can reduce reasoning “drift” and improve the model’s generalization, but may also alter the reasoning style. Whether to use few-shot prompts in the later training stage depends on specific applications and can be treated as a hyperparameter to tune.
💡

Conclusion

  • Research Contribution: The proposed STaR method generates reasoning datasets through self-iteration, which effectively improves model performance across various reasoning tasks. It enables the model to learn and refine its reasoning through self-generated outputs.
  • Limitations: STaR has certain limitations, such as requiring the initial model to already possess some reasoning ability. Tasks where guessing is easy (e.g., binary choices) often produce poor rationales that confuse STaR, and finding a way to filter out these bad explanations remains an open challenge.
      • Future work could explore more complex techniques to broaden its applications across more domains.
 
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant