Don't use compile rate to evaluate LLM vulnerability repair; it's gamed by non-repairs and dominated by evaluation setup, not model quality. Use change-aware metrics like diff_F1 as a first-pass screen before running actual tests.
This paper shows that compile rate—a common metric for measuring LLM-based code vulnerability repair—is unreliable because it's dominated by dataset artifacts rather than model quality, shifts dramatically with compiler flags, and even rewards broken patches.