For distillation to work well, the critic that evaluates student quality must be scaled proportionally with the teacher and should condition on generation steps—otherwise training can collapse or students fail to improve.
This paper improves Distribution Matching Distillation (DMD), a method for training fast student models from slower teacher models. The authors show that the critic network—which guides student training—needs to be scaled alongside the teacher and should be aware of generation steps to provide reliable training signals.