On-policy distillation succeeds only when the teacher model offers genuinely new capabilities beyond the student's training data and both models share compatible reasoning patterns—not just higher scores.
This paper investigates why on-policy distillation (a technique for training smaller AI models from larger ones) sometimes works and sometimes fails. The researchers found that success requires compatible thinking patterns between student and teacher models, plus genuinely new capabilities from the teacher.