Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Key Takeaway

Chain-of-thought reasoning traces appear interpretable but don't reliably encode step importance; this gap matters for process reward models and other techniques that assume reasoning text reflects functional significance.