Current research agents can generate valuable experiments but lack reliable judgment about which ones to prioritize—their rankings don't meaningfully improve experiment selection compared to simpler baselines.
This paper evaluates whether AI research agents can effectively judge which experiments are worth running by having six frontier LLMs propose minimum viable experiments for NLP papers, forecast their success, and rank them.