When comparing LLMs head-to-head, most leaderboard rankings don't have enough statistical power to be trustworthy. Use the paper's resolution ratio (q = N/N*) to check if a pairwise comparison is actually conclusive before relying on it.
This paper reveals that many pairwise comparisons on popular LLM leaderboards lack statistical power to detect real differences between models. The author shows that standard statistical calculators underestimate the sample size needed for paired comparisons by roughly 2x, leaving rankings unresolved—meaning we can't confidently say one model is better than another.