Language models can reason about component dependencies in small systems but become unreliable as complexity grows—and the reasoning cost is high (thousands of tokens per question), though formal semantics suggest this cost could be avoided with better approaches.
CordisBench is a benchmark testing whether language models can reason about how changes to software components affect other parts of a system. It uses a runtime called Cordis that manages dependencies and cleanup, asking models to predict what happens when components are modified or removed.