Lazy loaded image9.4 Prompting

9.4.1 Self-Refine: Iterative Refinement with Self-Feedback

Paper link: https://arxiv.org/abs/2303.17651

Research Background

The Self-Refine process mimics a three-step human-like approach of problem-solving, where the initial output from an LLM is refined through a series of iterative feedback steps. The biggest advantage of this method is that it requires no supervised training data, additional training, or reinforcement learning. A single LLM is used for generation, refinement, and feedback.

Research Method

⭐
Overall Structure
Generate → Self-feedback → Revise → Self-feedback → Revise → …
This approach combines a model’s instruction-following and evaluation abilities, which are necessary for the Self-Refine process. If the evaluation ability is insufficient for a specific task, the method will not be effective. However, if external evaluation is accurate, introducing external feedback can also make the process effective.
notion image
⭐
Self-Refine is an iterative self-refinement algorithm that alternates between two generation steps (self-feedback and revision). These steps work together to produce high-quality outputs.
Given an initial output generated by model M, it is input back into the same model to obtain feedback. Then, the feedback is again fed into the same model to improve the previously generated draft. This process is repeated for a specified number of iterations, or until the model determines that no further refinement is necessary.
As illustrated, given an input (0), Self-Refine first generates an output and sends it back to the same model M to obtain feedback (1). The feedback is then fed back into M, which refines the previously generated output (2). Steps (1) and (2) are repeated iteratively until the stopping condition is met. Self-Refine is instantiated with models such as GPT-3.5 and does not require human assistance.
notion image
Self-Refine algorithm:
notion image

Experiments

🔬
The study conducted experiments on three tasks (code optimization, sentiment transfer, and acronym generation), using different feedback generation methods, including:
  1. Self-Refine feedback
  1. Generic feedback: The feedback generated by the LLM is more general rather than highly specific. For example, in a code optimization task, the strategy might be “Improve the efficiency of the code” rather than “Avoid repeated calculations in the for loop.”
  1. No feedback: This removes the LLM-generated feedback step, simply passing the previous output back to the LLM for rewriting.
Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the feedback step of Self-Refine.
Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the feedback step of Self-Refine.
🔬
This study evaluated three tasks, analyzing how the results of Self-Refine change with different numbers of refinement iterations. below denotes the -th iteration. The figure shows that as the number of iterations increases, the quality of the generated responses improves, but the magnitude of improvement gradually diminishes.
Left: Iteration-wise score improvements. Early iterations significantly improve output quality, and scores generally keep improving with more iterations. Right: self-refine performance improvements with iterations. Most gains(∆) are in the initial iterations for both Code Optimization and Sentiment Reversal. The numbers re averaged over ChatGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).Most gains(∆) are in the initial iterations for both Code Optimization and Sentiment Reversal. The numbers re averaged over ChatGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).
Left: Iteration-wise score improvements. Early iterations significantly improve output quality, and scores generally keep improving with more iterations. Right: self-refine performance improvements with iterations. Most gains(∆) are in the initial iterations for both Code Optimization and Sentiment Reversal. The numbers re averaged over ChatGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).Most gains(∆) are in the initial iterations for both Code Optimization and Sentiment Reversal. The numbers re averaged over ChatGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).
In a code optimization task, the initial output (y0) had a score of 22.0. After three iterations (y3), the score increased to 28.8. Similarly, in a sentiment reversal task, the initial output scored 33.9, and after three iterations, it improved to 36.8.
These findings suggest that it is necessary to strike a balance between quality/improvement and the number of iterations. Multiple iterations introduce factors such as delay, cost, and speed limitations.

9.4.2 Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Paper link: https://arxiv.org/abs/2305.10601

Research Background

📌
When we handle new problems, we often make repeated adjustments and explorations before we can reach a final solution. During the exploration process, we generate some heuristic information that helps produce new discoveries. Therefore, a tree structure is needed for LLMs.
  • Tree-structured search is a general problem-solving process in AI.
  • Compared with LLMs, symbolic AI has limited generality due to its manually designed heuristic algorithms.
  • Hence, combining the generalization ability of LLMs with the advantages of tree structures may lead to better performance.

Method

notion image
⭐
ToT (Tree of Thoughts): The core idea of ToT is replacing the linear Chain of Thought (CoT) with a tree structure. Using GPT-4, branches are generated, and conditions are evaluated to decide which branches are suitable to extend or prune.
Key Steps:
  1. Task Decomposition (Manually Defined)
    1. The granularity should not be too fine. Otherwise, evaluation becomes difficult (e.g., generating a single token).
    2. The granularity should not be too coarse. Otherwise, it becomes hard to produce accurate results (e.g., generating an entire book).
  1. Generating “Thoughts”
    1. Multiple independent samplings: suitable for cases with large generation spaces.
    2. Generate all at once: suitable for smaller generation spaces, to avoid redundant problems.
  1. Evaluating the Feasibility of “Thoughts”
    1. CoT Scoring / Ranking.
    2. CoT Voting.
    3. Can be repeated independently multiple times, performing CoT Self-Consistency (CoT-SC).
  1. Tree Search
    1. BFS (Breadth-First Search): During each iteration, keep the top N best nodes.
        2. notion image
    2. DFS (Depth-First Search): During each iteration, limit the depth of the tree.
        3. notion image
Comparison between CoT, CoT-SC, and ToT
Item
CoT
CoT-SC
ToT
“Thought” Exploration
—
—
Has both DFS and BFS exploration methods
Effectiveness
★
★★
★★★
Token Consumption
★
★★★
★★★
Prompt Tuning Cost
★
★
★★
Applicable Scenarios
Complex tasks
Situations with limited correct labels
Complex tasks
LLM Capability Requirements
CoT
CoT
CoT, Evaluation
SFT (Supervised Fine-Tuning)
Not required
Not required
Not required

Experiment and Result

Model: GPT-4
Tasks: Three tasks that require searching or detailed planning
notion image

Application

✨
Game of 24
notion image
ToT Setup: Generate all possibilities at once + graded evaluation (sure / maybe / impossible) + BFS
notion image
✨
Creative Writing
notion image
ToT Setup: Multiple independent samplings + voting + BFS
Procedure:
  1. Generate 5 plans.
  1. Vote 5 times to select the best plan, then use that best plan to generate 5 segments.
  1. Vote 5 times again to select the best segment as the final result.
Performance:
notion image
The paper only evaluated logical coherence, and ToT performed better. Iterative-Refine also worked well for this task, and it can be combined with ToT.
✨
Mini crosswords
In Mini Crosswords, (a) how thoughts are proposed and aggregated in a priority queue for depth-first search (DFS), and (b) how a state is evaluated based on the possibility of filling in each remaining word clue, and pruned if any remaining clue is deemed not possible to fill by the LM. Then DFS backtracks to the parent state and explore the next promising thought for clue.
In Mini Crosswords, (a) how thoughts are proposed and aggregated in a priority queue for depth-first search (DFS), and (b) how a state is evaluated based on the possibility of filling in each remaining word clue, and pruned if any remaining clue is deemed not possible to fill by the LM. Then DFS backtracks to the parent state and explore the next promising thought for clue.
ToT Setup: Multiple independent samplings + generate all at once + graded evaluation (sure / maybe / impossible) + DFS
notion image

Research Summary

Although these experiments are relatively simple, they reflect abilities that are important in the real world. For example, the Game of 24 represents scientific exploration, creative writing represents the task of writing an article, and word puzzles represent optimization problems under specific constraints.
ToT Principle: It combines self-evaluation and considers the possibilities of each “thought step”, finding the correct solution from the entire thought tree.
Prev
9.3 o1 Techinical Paper
Next
9.5 LLM-as-Judge
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant