Aggregate accuracy metrics mask per-example instability in LLMs—models need per-example reliability evaluation, not just overall performance scores, to catch context-induced prediction flips.
This paper reveals that while large language models appear robust to irrelevant context when measured by overall accuracy, they actually flip predictions on individual examples unpredictably. Even meaningless text can shift model outputs, creating hidden reliability risks that aggregate metrics fail to capture.