Evaluating LLMs in interactive settings requires measuring both model performance and how users change their behavior in response—offline benchmarks and live experiments can disagree on which model is better.
When users interact with LLMs, they adapt their prompts based on model responses, changing what the model gets evaluated on. This paper shows that offline benchmarks and live A/B tests measure different things: offline tests compare models on the same prompts, while live tests compare them on user-adapted prompts. These can rank models differently and lead to conflicting deployment decisions.