AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.5.1.1 Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales
Paper link: https://arxiv.org/abs/2305.07095
Release date: May 11, 2023
Background
Recent studies on LLMs show that they can generate seemingly reasonable natural language texts (free-text rationales), which in turn can significantly improve their performance on leaderboards.
However, although these rationales appear effective in specific tasks, their actual usefulness for humans in real-world applications remains uncertain. In particular, it is still unclear whether humans truly benefit from these model-generated explanations or rationales when trying to understand problems and find correct answers. At present, evaluating whether such rationales are useful for humans is both expensive and difficult. In addition, the lack of effective metrics makes it challenging to assess these natural language rationales without human involvement.
Difference Between “Free-Text Rationalization” and “Chain of Thought”
Free-Text Rationalization (FTR): This method involves providing detailed explanations or justifications when answering a question, explaining why a particular answer is given. Its goal is to display the reasoning process while answering, making the response more transparent and easier to understand.
Features:
- Explanatory: It emphasizes justifying the correctness or reasonableness of an answer, explaining why it is correct.
- Educational: It helps users better understand complex concepts or the logic behind an answer.
- Applications: Suitable for contexts that require explanations or justifications to enhance credibility and understanding, such as education, legal advice, or complex scientific interpretations.
Chain of Thought (CoT): This is a method for solving problems in which the model demonstrates a step-by-step reasoning process from the question to the answer. It resembles a person’s internal dialogue while solving a problem, showing the intermediate reasoning steps.
Features:
- Stepwise: It breaks down problem-solving into sequential steps, with each step forming part of the reasoning process.
- Transparency: By revealing the CoT, it improves the clarity and interpretability of the model’s answers.
- Applications: Especially suitable for mathematical problems, logical reasoning tasks, or any complex question that requires multi-step analysis.
Do Rationales Help Humans?
The first part of the study aims to answer a fundamental question: do free-text rationales produced by LLMs actually help humans perform better on reasoning tasks? While prior work has shown that rationales improve model accuracy, it remains unclear whether those same explanations truly assist humans in making correct judgments.
To investigate this, the authors designed a human evaluation experiment using two datasets: StrategyQA and OpenBookQA. They selected several self-rationalizing LLMs (GPT-3, T5-3B, and T5-Large) to produce both an answer and a natural-language rationale.
Human Evaluation Setup
Given an input–output question pair (x, y), the model takes x as input and generates both the predicted answer and its corresponding rationale . A human predictor H is required to answer the question with two settings:
- H receives only the input x and produces an answer .
- H receives both x and the rationale , producing an answer .
Based on these results, the rationale’s usefulness is categorized as follows:
That is, if the human incorrectly solved the task before and corrects the answer after reading the rationale, it means the rationale is useful.
Findings and Correlation Analysis
The results revealed that only 15–20% of model-generated rationales were useful, while 30–55% were misleading, which doesn’t help humans to select the correct answer. The remaining cases were “unsure”, where the rationale made no measurable difference. This means that even though models can articulate seemingly reasonable explanations, most of these rationales fail to help humans reason better.
Correlation Analysis: Why Rationales Fail to Help
The authors analyzed whether existing model-level metrics could predict which rationales would be useful to humans. They measured two types of statistical correlations:
- Theil’s U (Uncertainty Coefficient): to test the correlation between task accuracy (how well the model itself answers) and human utility (how well the rationale generated by the model can help humans choose the correct answer).
- Correlation Ratio (η): to test the correlation between BERTScore (similarity between generated and gold rationales) and human utility. The gold rationales here are human-annotated reference explanations used for comparison.
Across both datasets, the correlations were extremely weak:
- Task accuracy vs. human utility: U ≈ 0.005–0.111
- BERTScore vs. human utility: η ≈ 0.002–0.055
These near-zero values indicate that neither the model’s correctness nor the resemblance of its rationale to human-written explanations reliably predicts whether the rationale will be helpful to humans.
From this first experiment, we find that rationales that make models appear more transparent or capable on benchmarks do not necessarily improve human reasoning accuracy. This finding challenges the assumption that better model explanations automatically lead to better human understanding, motivating deeper analysis in the next section: What Makes Rationales Useful?
What Makes Rationales Useful?
This study analyzes how different features of rationales relate to human utility using the StrategyQA dataset. They identified eight key properties, grouped into four conceptual axes that describe both the form and content of a rationale:
To quantitatively estimate which properties contribute to human usefulness, the authors applied a Generalized Linear Mixed-Effects Model (GLMEM), which is a statistical model that measures how individual properties and their pairwise interactions predict human performance.
To be specific, the GLMEM is a logistic regression predicting whether a rationale helps humans (1 or 0). Each property (like validity or novelty) is a labeled feature, known for each rationale. The model tests which combinations of properties increase the chance that humans answer correctly after seeing the rationale, while controlling for question difficulty, model type, and human prior knowledge.
Findings: Human Utility Depends on Balanced, Informative Rationales
The analysis revealed several insights:
- No single property alone consistently predicts usefulness
- When properties interact in combination, utility increases significantly. Without leaking the answer, rationales that were grammatical + novelty, or conciseness + novelty, tended to help humans the most.
In short, humans are robust to minor stylistic issues like redundancy or mild hallucinations, but are misled when rationales align too strongly with incorrect model answers. Therefore, good rationales are not just well-written; they must be informative, factually grounded, and independent of model bias.
Human Generalization Experiments
In the third part, the author discusses whether useful rationales help humans generalize knowledge beyond the original question. This part of the study focuses on measuring knowledge transfer, whether seeing a good rationale enables a person to solve rephrased, counterfactual, or analogous questions that were never seen before.
Experiment Setup: Generalization Question Types
The StrategyQA dataset is used and create several generalized question sets . Each set contains reworded or modified versions of the original question.
There are three types of generalization questions:
- Paraphrased questions – non-trivial rewordings that preserve meaning but change phrasing.
- Counterfactual questions – similar reasoning but with the opposite answer.
- Analogical questions – follow similar reasoning steps but use different entities or contexts.
This setup allows them to assess whether rationales improve human (and model) understanding beyond just the specific task instance.
Human Evaluation Procedure
Human evaluators again participated in a two-step process, similar to the earlier human-utility test:
- Without rationale: answered the new question to produce .
- With rationale: was then shown the original rationale (from the base question, not rewritten) and asked to answer again, giving .
This design tests whether the reasoning expressed in the rationale for one question can transfer to another similar question.
Results: Transferability of Useful Rationales
The findings confirmed a strong link between human utility and generalization ability:
- Rationales previously labeled as useful also led to higher gains when applied to new, related questions across all three generalization types.
- Not-useful rationales often reduced accuracy, indicating that misleading reasoning actively harms knowledge transfer.
- The degree of generalization improvement was consistent across rephrased, counterfactual, and similar-reasoning questions, showing that usefulness is not tied to surface similarity but to genuine reasoning quality.
Through this experiment, the authors demonstrate that human-utility is not just about short-term correctness. It reflects the reasoning quality that generalizes.
Automating Human-Utility Measurement and Improvement
While human evaluation provides a clear measure of how helpful model-generated rationales are, it is also time-consuming, expensive, and difficult to scale. To solve this, the authors propose an automatic metric that estimates human utility without requiring human annotators. The authors introduce GEN-U (Generalization-based Utility), an automatic evaluation framework that mirrors the human experiments but uses LLMs.
GEN-U Formula
For a given input–output pair , let there be a set of related or paraphrased (generalization) questions:
We define:
- : a language model that takes only as input and predicts .
- : a language model that takes both and the rationale (generated by a self-rationalizing model ) as input and predicts .
Then, the GEN-U score for a single example is given by:
Here:
- is an indicator function that equals 1 if the condition is true, and 0 otherwise.
- takes the most frequent value among the results (similar to majority voting).
Intuitive Explanation
- If the model answers correctly after seeing the rationale (), then the score depends on whether it could already answer correctly without the rationale:
- If it couldn’t before but can now → GEN-U = 1 (useful rationale)
- If it could already → GEN-U = 0 (redundant rationale)
- If the model answers incorrectly even after seeing the rationale → GEN-U = −1 (not useful)
- Remember that for each input–output pair , there is a set of paraphrased questions, each of which produces a GEN-U sub-score, where:
- 1 → rationale was useful
- 0 → rationale was redundant (already solvable)
- −1 → rationale was not useful
An example GEN-U sub-score could be:
Generalization Question | Sub-score |
#1 | 1 |
#2 | 1 |
#3 | −1 |
#4 | 1 |
#5 | 0 |
To get one final GEN-U score for that original question, we take the MODE, i.e. the value that appears most often. In this example, “1” appears most frequently, so .
Validating GEN-U Against Human Judgments
To confirm that GEN-U truly reflects human evaluation, the authors compared GEN-U scores with the human-utility labels collected from earlier experiments on the StrategyQA dataset.
Each rationale in the dataset has:
- A human-utility label (whether the rationale actually helped humans answer correctly), and
- Several automatic metric values, including GEN-U, task accuracy, and BERTScore.
The analysis measures how well each automatic metric aligns with human judgments. Specifically, the authors compute correlation coefficients between human-utility labels and each metric:
The results show that GEN-U exhibits the strongest alignment with human evaluations. This experiment validates that GEN-U can be used as an evaluation model to replace humans.
Improving Model Rationales Using GEN-U Feedback
After validating GEN-U, the authors explored whether it could be used as a training signal to improve LLM rationale generation. They employed the Quark algorithm (Lu et al., 2022), a reinforcement-style optimization framework where the model is rewarded based on an external feedback signal, and in this case, the GEN-U score.
Steps:
- Start with a self-rationalizing model (T5-based LMs).
- For each rationale it generates, compute a GEN-U score based on how much that rationale improves model performance on generalized questions.
- Use this GEN-U score as a reward signal to update the model, resulting in an improved model .
Result:
- The updated model produces more useful and fewer misleading rationales across both StrategyQA and OpenBookQA.
- For example, the proportion of useful rationales increases by about 2%, while misleading ones decrease correspondingly.
Conclusion
This work examines the human utility of free-text rationales, how well they help people solve reasoning tasks. Extensive human evaluations show that current LLM-generated rationales are often not truly helpful, and that common automatic metrics poorly reflect human benefit. The study finds that generalization ability serves as a strong proxy for human utility and can be used as a reward signal to improve models.
Future work should explore fine-grained rationale properties and extend human-utility evaluation to more challenging tasks, such as closed-book question answering.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...