When training student models on their own generated data, length inflation and repetition can cause sudden training collapse—StableOPD fixes this with divergence constraints and mixture distillation, making on-policy distillation more reliable.
This paper identifies a critical failure mode in on-policy distillation (OPD)—a training technique where student models learn from teacher models using their own generated data. The problem: as training progresses, generated sequences become unexpectedly long and repetitive, causing training data to become dominated by truncated examples and destabilizing the entire training process.