You can adapt a frozen language model to new rewards at inference time by carefully controlling how much you perturb its hidden states—using Fisher information to measure and limit distributional changes prevents quality degradation.
This paper introduces MISVO, a technique for steering frozen language models at test time by adding vectors to hidden states while minimizing unwanted changes to output quality. Using Fisher information geometry, the method penalizes interventions that distort the token distribution, enabling efficient reward optimization without retraining the model.