Multimodal AI models can follow general lab procedures but fail at the precise, action-grounded perception needed for actual chemical lab automation—they can't reliably identify where to act, what equipment is present, or how to sequence complex procedures safely.
BEAKER is a benchmark with 1,000 expert-curated samples for evaluating multimodal AI models as 'embodied brains' in self-driving chemical labs. It tests whether AI can perceive lab scenes, reason about experiments, and execute actions safely—covering tasks like identifying where to act, planning movements, recognizing equipment and hazards, and sequencing steps.