Teachers in distillation can fail to correct student errors—adding explicit critique and revision cycles recovers reliable supervision and improves learning on complex reasoning tasks.
This paper addresses a flaw in on-policy distillation where teachers blindly follow student errors instead of correcting them. CriticOPD fixes this by having the teacher critique mistakes, allowing students to revise their solutions and learn from corrections. Tests on math reasoning tasks show significant improvements, especially on harder problems.