You can replace expensive LLM judges with one-time program generation that's faster, cheaper, and fully inspectable—achieving comparable accuracy to frontier models while costing just $0.17-0.49 per topic instead of per-document LLM calls.
This paper proposes ExecJudges, a method that uses LLMs to write readable programs instead of making individual relevance judgments. These programs use lexical matching and NLP tools to automatically label documents, eliminating expensive per-document LLM calls while remaining interpretable and reproducible.