Decision models can match or exceed generative LLM ranking quality through on-policy distillation, while being orders of magnitude faster—enabling practical deployment for retrieval tasks.
DecisionRank trains a decision model to rank documents by predicting scores directly, avoiding slow autoregressive decoding. It uses a novel on-policy distillation method where a generative teacher LLM provides feedback on the student's predicted rankings, enabling a 1.7B model to outperform its 27B teacher while running 24× faster.