Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning

Yuanteng Chen, Zhilei Liu, Peisong Wang, Yuantian Shao, Chuangyi Li et al.|September 22, 2026arXiv

Key Takeaway

On-policy distillation recovers reasoning capabilities in ultra-low-bit quantized models by training on the model's own generated outputs rather than fixed data, fixing the exposure bias problem that causes long-form reasoning to fail.

Summary

This paper tackles a critical problem in quantized language models: when you compress models to very low precision (under 3 bits), they lose the ability to do math and coding tasks because errors compound during long generation.

training efficiency reasoning

Key Terms

quantization-aware-training exposure-bias on-policy-learning autoregressive-decoding knowledge-distillation