When distilling guided diffusion models, directly matching teacher and student predictions can backfire if they have asymmetric access to information—you need to supervise different prediction branches separately to avoid antagonistic error dynamics.
This paper identifies a critical flaw in how diffusion models are distilled when using classifier-free guidance (CFG). The authors show that naive matching of teacher and student predictions can fail when the teacher has access to information the student doesn't, causing errors in different prediction branches to work against each other.