When adapting language models at test time using self-generated labels, you should weight updates based on how reliable the voting signal is for each prompt—this prevents the model from learning from noisy pseudo-labels.
This paper addresses a key problem in test-time reinforcement learning: how reliable are the self-generated labels used to adapt language models? The authors analyze majority-vote pseudo-labels, identify sources of error, and propose PC-Dr.DPO, a method that makes the model more cautious when updating from unreliable prompts. They validate their approach across math and reasoning tasks.