When using SWE-bench benchmarks to evaluate LLM coding abilities, be aware that over 1 in 10 test cases have misaligned problem statements and solutions—PAIChecker can automatically identify these problematic cases to improve benchmark reliability.
This paper identifies a critical quality issue in SWE-bench-like benchmarks used to evaluate AI coding abilities: 13.6% of PR-Issue pairs are misaligned, meaning the issue description doesn't actually match the code changes.