Language models can sound confident about temporal patterns in data without doing proper statistical testing—this benchmark reveals they fail mainly at selecting relevant evidence, not at running tests, and a structured retrieval-then-test approach significantly improves accuracy.
This paper introduces STRATA, a benchmark for evaluating language models on temporal clinical hypothesis testing. Rather than just grading final answers, it evaluates every step: choosing which patient records matter, selecting the right statistical test, and interpreting results.