Security agent evaluations should measure cost per success, not just peak performance—offensive and defensive tasks have different scaling properties, and cheaper models can match expensive ones on some tasks when evaluated fairly.
This paper evaluates AI security agents not just by success rate, but by cost-efficiency—measuring how much computation and tool usage is needed to succeed. Testing on offensive hacking challenges and defensive security investigation tasks, the authors find that offensive tasks benefit from more reasoning budget, while defensive tasks depend more on smart tool selection than raw compute.