IRT models commonly used in AI benchmarks can produce unreliable results when applied to typical benchmark conditions (few models, many items, skewed distributions), so practitioners need to validate their specific setup before trusting IRT-based rankings or diagnostics.
This paper examines whether item response theory (IRT)—a statistical method borrowed from human testing—reliably works for AI benchmarking.