Autonomous research systems need standardized benchmarks with clear success criteria; this work shows simple baselines often outperform sophisticated methods, suggesting current auto-research has significant room for improvement.
Auto-Research Arena is a benchmark for evaluating autonomous research systems that discover improvements in language modeling. It separates what counts as a valid result (admissibility) from what measures progress (targets), and tests seven different research methods across nine objectives like model efficiency and training speed.