Don't trust matched evaluation scores for coding agents—they hide failures introduced by command serialization and parsing.
This paper reveals that standard evaluation metrics for LLM coding agents can hide critical failures in command execution. By testing how Bash commands survive serialization and parsing in different system configurations, the authors show that matched scores mask up to 73% of actual failures—failures introduced not by the model but by how its output is processed.