RationaleRM

English | 中文 Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models [📄 Paper] • [🤗 Dataset] • [📜 Citation] Outcome Accuracy vs Rationale Consistency: Rationale Consistency effectively distinguishes frontier models and detects deceptive alignment 📖 Overview RationaleRM** is a research project that investigates how to align not just the *outcomes* but also the *reasoning processes* of reward models with human judgments. We discover that generative reward models (GenRMs) and LLM-as-a-Judge exhibit **Deceptive Alignment** issues — models may reach the same final result as humans through superficial or even incorrect reasoning processes. To address this, we propose the **Rationale Consistency** metric, which measures the alignment between the model's reasoning process and human judgment rationales. We also design the **MetaJudge** framework to compute this metric: it decomposes human and model rationales into atomic units, then performs strict one-to-one semantic matching to precisely quantify their consistency. Core Contributions:** 🔍 **MetaJudge Framework**: Decomposes human rationales into atomic units and uses LLMs for strict one-to-one semantic matching 📊 **Rationale Consistency Metric**: Effectively detects deceptive alignment and distinguishes frontier models (e.g., GPT-5 or Gemini 3 Pro) 🛠️ **Hybrid Reward Training**: Combines rationale reward (Average Precision) and outcome reward to prevent "rationale degeneration" 🏆 **SOTA Performance**: Achieves best results on RM-Bench (87.1%) and JudgeBench (82.0%) 🚨 Key Finding: The Deceptive Alignment Trap We evaluated 19 frontier models and found two critical flaws when relying solely on outcome accuracy: Outcome Accuracy Cannot Distinguish Frontier Models In the green region, although multiple models achieve similar outcome accuracy, rationale consistency clearly distinguishes stronger models (such as GPT-5, o3, Gemini 3 Pro) from weaker ones (such as Claude 3.5, GPT-4.1). Outcome Accuracy Canno