How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai et al.|November 8, 2026arXiv

Key Takeaway

On-policy distillation tolerates significant training-inference mismatch from quantization and asynchronous rollouts without quality loss, enabling 1.6× faster training with FP8 precision while maintaining performance.

Summary

This paper studies how much training-inference mismatch on-policy distillation (OPD) can tolerate when using quantized inference and asynchronous rollouts to reduce training costs.

training efficiency

Key Terms

on-policy-distillation training-inference-mismatch quantization asynchronous-rollouts kv-cache