You can train models at inference time without labels by treating agreeing and disagreeing predictions differently—distilling correct-looking outputs while penalizing confident mistakes—achieving performance comparable to supervised methods.
TTPO enables language models to improve at test time without ground-truth labels by using an asymmetric training approach: it distills rollouts that agree with majority-vote predictions while penalizing disagreeing ones. This works even when the majority vote is wrong, because incorrect predictions tend to be rejected by the model anyway.