Benchmarks require ongoing maintenance as models improve—not just fixing bugs in tasks, but also revalidating resource constraints and evaluation assumptions that can materially change what capabilities are actually being measured.
This paper addresses how benchmarks for coding agents become outdated as models improve and infrastructure changes. The authors introduce a continuous validation workflow for maintaining benchmarks, apply it to Terminal-Bench 2.0, and find issues in ~30% of tasks.