🔗9.4.1.2 ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

Paper link: https://arxiv.org/abs/2212.07919
Release date: December 15, 2022
⭐
This paper proposes ROSCOE, a framework of automatic metrics designed to evaluate the quality of step-by-step reasoning generated by large language models (LLMs). Unlike traditional evaluation methods that focus only on the final answer or surface-level text similarity, ROSCOE aims to directly assess the reasoning process itself, including whether the reasoning is grounded in the input, logically consistent, informative, and linguistically coherent. The goal of the paper is not to improve LLM reasoning, but to provide an evaluation tool for diagnosing reasoning quality.
Note that this work was published in the early stage of chain-of-thought (CoT) reasoning for LLMs. The models, tasks, and reasoning patterns considered are relatively simple compared to modern LLMs. Therefore, its assumptions and metrics may not fully capture the behavior of today’s more powerful LLMs.

Background

LLMs show improved performance on many reasoning tasks when prompted to generate step-by-step explanations CoT before producing a final answer. These reasoning steps enhance model interpretability and make verification more transparent. However, despite their importance, there is still a lack of reliable automatic evaluation methods for assessing the correctness of the reasoning process itself, independent of the final answer. Most existing metrics focus on surface-level text similarity or final-task accuracy and cannot effectively detect logical errors, hallucinations, missing steps, or inconsistencies within reasoning chains. This gap motivates the need for specialized metrics to systematically evaluate the quality of step-by-step reasoning.

Method - Reasoning Error Taxonomy

Before designing evaluation metrics, the authors first define what kinds of mistakes can occur in step-by-step reasoning. The paper defines the following nine reasoning error types:
  1. Grammar: The reasoning contains ungrammatical language.
  1. Factuality: A stated fact contradicts the source context.
  1. Hallucination: The reasoning introduces information not present in the source.
  1. Redundancy: The reasoning contains correct but unnecessary steps that do not contribute to solving the problem.
  1. Repetition: A step restates a previous step without adding new information.
  1. Missing Step: A required intermediate inference is omitted.
  1. Coherency: Different steps contradict each other or do not form a consistent chain. Example: First stating A > B, later assuming B > A.
  1. Commonsense: The reasoning lacks basic world knowledge needed to connect steps.
  1. Arithmetic: The mathematical computation is incorrect.
Each ROSCOE metric is later constructed to detect one or more of these categories.

Method - ROSCOE Metric Framework

ROSCOE is a metrics that evaluate step-by-step reasoning through semantic comparison between the input context and the generated reasoning. The basic idea is to convert reasoning steps into vectors, then evaluate their relationships using similarity and consistency measures.
⭐
Sentence Embedding
ROSCOE encodes
  • each source sentence
  • each reasoning step
into dense vectors using a SimCSE sentence embedding model (fine-tuned on reasoning data in this work). All subsequent computations operate in this embedding space. This allows ROSCOE to compare sentences by semantic meaning rather than surface word overlap.
⭐
Core Alignment Formula
For each reasoning step , ROSCOE computes its semantic alignment to the source as:
This measures how well a reasoning step is grounded in the input by finding the most semantically similar source sentence. The result is a value in , where higher means stronger grounding.
Using this step-level alignment vector, multiple interpretable sub-metrics are constructed.
⭐
Examples of Key Sub-Metrics
Rather than collapsing everything into one score, ROSCOE provides 18 fine-grained metrics, each targeting different reasoning properties. The most representative ones include:
  • Faithfulness-Step
      • Measures whether reasoning steps are grounded in the input. Low values indicate hallucinations or misuse of source information.
  • Informativeness-Step (Info-Step): A bidirectional alignment between source and reasoning that measures how much of the source information is actually used in the reasoning. Low values indicate missing key information.
  • Repetition (step-to-step alignment): Measures semantic similarity among reasoning steps to detect redundant or repeated reasoning.
In addition to these semantic-alignment metrics, ROSCOE also includes:
  • Logical Inference metrics (based on NLI models) to detect contradictions,
  • Language Coherence metrics (based on perplexity and grammar classifiers) to evaluate fluency.
⭐
Four Metric Perspectives
All of the 18 ROSCOE metrics are grouped into four perspectives:
  1. Semantic Alignment (SA): grounding and information usage,
  1. Semantic Similarity (SS): overall chain-level similarity and repetition,
  1. Logical Inference (LI): internal and source-level consistency,
  1. Language Coherence (LC): fluency and grammar.
Rather than producing a single global score, ROSCOE outputs a set of interpretable diagnostic signals, allowing users to understand why a reasoning chain fails.

Method - Dataset Construction for Evaluation

To evaluate whether ROSCOE can truly distinguish good reasoning from bad reasoning, the authors construct two complementary types of datasets: synthetic diagnostic datasets and human-judged datasets. Together, these enable both controlled testing and realistic evaluation.
⭐
1. Diagnostic (Synthetic) Datasets
These datasets are built from benchmarks that contain gold step-by-step reasoning (e.g., EntailmentBank, ProofWriter, MATH, AQUA, ASDIV, EQASC). The authors generate faulty reasoning chains by applying rule-based perturbations derived from the error taxonomy, such as:
  • inserting arithmetic mistakes,
  • removing a required step,
  • negating a statement,
  • repeating a step,
  • introducing hallucinated content.
Each original reasoning chain is paired with a corrupted (injected error) version, and a binary label is assigned:
  • 0 = clean (no error),
  • 1 = corrupted (with injected error).
This setting allows testing of whether a metric can separate correct reasoning from corrupted reasoning under controlled conditions.
⭐
2. Human-Judged Datasets
To avoid reliance on synthetic errors alone, the authors also evaluate ROSCOE on human-annotated data. They select several real-world reasoning benchmarks (e.g., GSM8K, DROP, ESNLI, COSMOS-QA, SemEval) and generate step-by-step reasoning using GPT-3 with few-shot prompting.
Each generated reasoning chain is then evaluated by annotators using the nine error types defined in the taxonomy. The annotations provide human ground-truth labels for different reasoning failures. These labels are later used to test how well ROSCOE correlates with actual human judgments of reasoning quality.
Purpose of Using Two Dataset Types
  • Synthetic datasets test whether ROSCOE can detect specific, controlled reasoning errors.
  • Human-judged datasets test whether ROSCOE aligns with real human perception of reasoning correctness.

Experiment and Results

The paper evaluates ROSCOE using two sets of experiments corresponding to two result tables: one on synthetic diagnostic datasets and one on human-judged datasets. In both cases, performance is measured using Somers’ D rank correlation, which quantifies how well a metric can distinguish correct from incorrect reasoning. Higher values indicate better discrimination ability.
⭐
Results on Diagnostic (Synthetic) Datasets
The table reports correlations between evaluation metrics and injected reasoning errors across six datasets (e.g., EntailmentBank, MATH, ProofWriter, AQUA, EQASC, ASDIV).
Somers’ D correlation of different metrics on six Diagnostics datasets.
Somers’ D correlation of different metrics on six Diagnostics datasets.
Key observations:
  • ROSCOE outperforms all baseline metrics on every dataset.
  • Semantic Alignment (ROSCOE-SA) and Semantic Similarity (ROSCOE-SS) achieve near-perfect correlations (often > 0.9), showing that they can almost perfectly separate clean reasoning from corrupted reasoning.
  • Baseline metrics typically achieve low correlations (≈ 0.1–0.4), indicating that surface-level text similarity is insufficient for detecting reasoning errors.
Under controlled conditions with known injected errors, ROSCOE is highly sensitive to reasoning corruption, while traditional metrics are not sensitive to this task.
⭐
Results on Human-Judged Datasets
The table below evaluates metrics against human expert annotations on five real-world reasoning benchmarks (e.g., GSM8K, DROP, ESNLI, COSMOS-QA, SemEval).
Somers’ D correlations of metrics with human judgement.
Somers’ D correlations of metrics with human judgement.
Key observations:
  • ROSCOE again achieves the highest correlations with human judgments across nearly all datasets.
  • Some baseline metrics even show negative correlations, meaning they sometimes assign higher scores to worse reasoning due to overemphasis on fluency and surface similarity.
  • ROSCOE-SA and ROSCOE-SS are the most stable top performers, demonstrating strong alignment with how humans judge reasoning quality.
  • Correlations on human data are lower than on synthetic data, reflecting the greater difficulty and subjectivity of real-world reasoning evaluation.
The paper further analyzes how ROSCOE behaves under different conditions to understand its reliability and limitations. The analysis focuses on three key aspects: sensitivity to error severity, threshold stability across datasets, and metric specialization.
⭐
Sensitivity to Error Severity
The authors study whether ROSCOE can reflect different levels of reasoning corruption, not just the presence of an error. They progressively inject 1, 2, and 3 errors into the same reasoning chains and observe how scores change.
They find that ROSCOE Semantic Alignment (SA) and Semantic Similarity (SS) scores decrease monotonically as more errors are injected. It shows that ROSCOE provides a graded signal of reasoning quality, rather than only a binary pass/fail judgment. In contrast, many baseline metrics fluctuate irregularly and fail to reflect increasing error severity.
 
Sensitivity of selected metrics on Somers’ D by injecting levels of error into reasoning steps.
Sensitivity of selected metrics on Somers’ D by injecting levels of error into reasoning steps.
⭐
Cross-Dataset Threshold Instability
The authors investigate whether a single fixed threshold (e.g., “score < 0.8 means bad reasoning”) could be used across all datasets.
Box-and-whisker plots of interquartile ranges of scores, for perturbations and reference-free metrics with strong Somers’ D values.
Box-and-whisker plots of interquartile ranges of scores, for perturbations and reference-free metrics with strong Somers’ D values.
It shows that within a single dataset, clean and corrupted reasoning are well separated. However, across different datasets, score distributions shift significantly. So, ROSCOE is best used for relative comparison within a dataset, and dataset-specific calibration is required for absolute quality judgments.
⭐
Specialization of Different ROSCOE Components
Relative frequency of strong score-perturbation correlations, measured as the proportion of datasets in which each score-perturbation pair achieves a Somers’ D value in the top 10%, using reference-free ROSCOE scores with fine-tuned SimCSE embeddings.
Relative frequency of strong score-perturbation correlations, measured as the proportion of datasets in which each score-perturbation pair achieves a Somers’ D value in the top 10%, using reference-free ROSCOE scores with fine-tuned SimCSE embeddings.
Different ROSCOE components respond to different types of errors:
  • SA and SS: most effective for hallucination, repetition, and missing-step errors.
  • Logical Inference (LI): best at detecting contradictions.
  • Language Coherence (LC): mainly reflects grammar and fluency, but is weak for logical or arithmetic correctness.
Implication: There is no single universal reasoning score. ROSCOE should be treated as a diagnostic toolbox, where different metrics are selected based on the target error types.

Limitation and Conclusion

Key Limitations:
  • ROSCOE is less effective for long, purely symbolic, or highly abstract reasoning where late steps no longer semantically resemble the input.
  • The framework is optimized for short, natural-language CoT reasoning, typical of early LLMs.
  • Absolute score thresholds are unreliable without domain-specific calibration.
Conclusion: ROSCOE is an early but influential framework for automatically evaluating step-by-step reasoning in LLMs. By combining a reasoning error taxonomy with embedding-based, interpretable metrics, it significantly improves over traditional text similarity measures. While effective for short natural-language reasoning, its limitations on long and symbolic reasoning highlight the need for newer evaluation methods for modern LLMs.
 
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...
Article List
LLM Learning Roadmap
✨ Awesome-Anything
Lazy loaded image Awesome-AI-Tutorials
Lazy loaded image Awesome-AI-Researchs
Lazy loaded image Awesome-GitHub-Repos
Lazy loaded image Awesome-AI-Examples
🖼️ Digital Image Processing
Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
Lazy loaded image DIP II Image Denoising
Lazy loaded image DIP III Edge Detection
Lazy loaded image DIP IV Digital Half-toning
Lazy loaded image DIP V Geometric Image Modification
🍃 LLM Components
Lazy loaded image 1.1 Tokenization
Lazy loaded image 1.2 Embedding
Lazy loaded image 1.3 Attention
Lazy loaded image 1.4 FFN, Residual Addition, and LN
Lazy loaded image 1.5 Positional Encoding
Lazy loaded image 1.6 Architecture & Decoding Policy
🌱 LLM Pre-training
Lazy loaded image 2.1 Pre-training Definition
Lazy loaded image 2.2 Pre-training Data
Lazy loaded image 2.3 Pre-training Workflow
Lazy loaded image 2.4 Continued Pre-training (CPT)
Lazy loaded image 2.5 Pre-training Evaluation
☘️ LLM Post-Training
Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
Lazy loaded image 3.2 Reinforcement Learning
Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
Lazy loaded image 3.5 Direct Preference Optimization (DPO)
🍀 LLM Popular Models
Lazy loaded image 4.1 BERT & Variants
Lazy loaded image 4.2 PaLM Series
Lazy loaded image 4.3 GPT Series
Lazy loaded image 4.4 LLaMA Series
Lazy loaded image 4.5 GLM Series
Lazy loaded image 4.6 Qwen Series
Lazy loaded image 4.7 DeepSeek Series
Lazy loaded image 4.8 MOE Series
Lazy loaded image 4.9 Other
🪴 LLM Applications
Lazy loaded image 5.1 Prompt Engineering
Lazy loaded image 5.2 LLM-based Agents
Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
🌿 LLM Optimization
Lazy loaded image 6.1 GPU Usage Analysis
Lazy loaded image 6.2 Latency Analysis for Inference
Lazy loaded image 6.3 FlashAttention
Lazy loaded image 6.4 PagedAttention
Lazy loaded image 6.5 Packing for Efficient Training
Lazy loaded image 6.6 PEFT
🌾 LLM Compression
Lazy loaded image 7.1 Quantization
Lazy loaded image 7.2 Pruning
Lazy loaded image 7.3 Knowledge Distillation
🌵 LLM Hands-on Practice
Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
🌴 LLM Must-read Papers
Lazy loaded image 9.1 Technical Reports Explained
Lazy loaded image 9.2 MedLLM
Lazy loaded image 9.3 o1 Techinical Paper
Lazy loaded image 9.4 Prompting
Lazy loaded image 9.5 LLM-as-Judge
🌳 LLM Q&A
Lazy loaded image 10.1 Why GRPO training loss starts at 0
Lazy loaded image 10.2 VLM-R1 Reproduction Results
Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
Lazy loaded image 10.4 Left Padding or Right Padding?
Lazy loaded image 10.5 What is QK-Norm?
Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
Lazy loaded image 10.7 Dropout and DropPath
🐝 VLM Image Encoders
Lazy loaded image 11.0 Introduction
Lazy loaded image 11.1 Vision Transformer
Lazy loaded image 11.2 CLIP
Lazy loaded image 11.3 BLIP
Lazy loaded image 11.4 EVA
Lazy loaded image 11.6 BLIP-2
Lazy loaded image 11.7 SigLIP
📝 MISC.
🤔 Large Language Models Pass the Turing Test.
Lazy loaded image OpenAI ChatGPT Progression
⏳ Claude Sonnet 4 Test with Sandtris
🔍 What DeepSeek-R1's Surge Teaches Us About SEO
💰 CivitAI’s Payment Issue
🎓 arXivSub, an Academia Assistant