Current metrics for evaluating user simulators are unreliable and can even prefer broken behavior; a comprehensive evaluation approach checking fidelity at multiple granularities is essential for building trustworthy simulators.
This paper evaluates how well we can assess the quality of user simulators—AI systems that mimic human users to test interactive agents.