LLMs fail to satisfy the law of total probability: estimates from fine-grained subpopulation prompts aggregate differently than direct population estimates, suggesting models have relevant knowledge but don't reliably combine it.
This paper tests whether large language models follow basic probability rules when making predictions. Researchers partition populations into subgroups, prompt models to estimate statistics for each subgroup, then aggregate those estimates back up.