By performing Bayesian inference in a low-dimensional parameter subspace rather than the full network, you can efficiently quantify reward model uncertainty and select informative preference queries without the computational overhead of full posterior inference.
This paper presents PreferenceEKF, a method for efficiently learning reward models from human preferences in RLHF. Instead of tracking uncertainty across all neural network parameters (which is computationally expensive), it uses an extended Kalman filter to track uncertainty in a low-dimensional subspace, enabling faster and more scalable active learning of reward models.