LLM-based semantic query engines are non-deterministic even at temperature 0—benchmark comparisons based on single runs can be misleading, and you need multiple executions or voting strategies to get reliable results.
This paper reveals that semantic query engines using LLMs produce inconsistent results across multiple runs, even at temperature 0. By re-executing 1.2 million model calls across three benchmarks, the authors show that 0.07-10.2% of individual rows change their output between runs, and up to 66% of final answers differ.