You don't need the most expensive LLM to judge citation quality—cheaper models match frontier models on accuracy—but all judges have directional biases that must be calibrated before using them as reward signals in AI training.
This paper evaluates which LLM judges are suitable for scoring citation quality in AI research systems. Researchers tested 8 different LLMs on 1,248 citation evaluations and found that cheaper models like GPT-4-mini perform comparably to expensive frontier models, but all judges have hidden biases in false positive/negative rates that could distort AI training if not addressed.