When fine-tuning RL policies, pretraining Q-functions on offline data often doesn't help; instead, initializing Q-functions from diverse policy rollouts works better and is simpler.
This paper challenges the conventional wisdom that Q-functions should be pretrained for online RL fine-tuning. The authors show that naive Q-function pretraining often doesn't help because it targets the wrong policy, then propose IPE—a method that uses diverse pretrained policies to bootstrap Q-function learning, achieving 1.26x better performance than standard pretraining.