Stop benchmarking AI on what it can do at its best; measure instead how consistently it does the same thing when asked the same question twice. This precision metric better predicts real-world system reliability and guides whether you need better rules or a better model.
This paper argues that AI system quality should be measured by precision (consistency of outputs across repeated requests) rather than capability (best-case performance). Using a marksman analogy, the author shows that frontier models have saturated accuracy but differ in output reliability—how tightly grouped their responses are.