By analyzing how agents solve problems—not just whether they succeed—you can evaluate SWE agents on fewer benchmark tasks while still accurately predicting full-benchmark performance.
This paper proposes PTA-IRT, a framework for efficiently evaluating software engineering agents on benchmarks by using execution trajectories (exploration paths, code edits, test results) alongside pass/fail outcomes. Instead of running every task, it selects representative tasks and uses trajectory data to better estimate how well agents would perform on the full benchmark.