LLM benchmarks often hide the real problem: even when LLMs get answers right on average, they're much less reliable than humans, with larger errors and higher variability—critical flaws for any real-world deployment.
This paper challenges claims that LLMs match human expert performance by testing both on a real data analysis coding task. The researchers found that human experts outperformed a frontier LLM on average and showed more consistent, reliable results—revealing that standard benchmarks miss critical reliability issues that matter in high-stakes work.