Pretrained language models can be efficiently adapted for robot control through simulation-based training alone, enabling strong real-world performance and compositional generalization without collecting real robot data.
ZAP is a robot control system that combines a pretrained vision-language model with simulation-based training to enable robots to understand tasks and execute them without real-world training data. The approach achieves 96.7% success on real-world manipulation tasks while being 3x faster and 17x cheaper than GPT-4 Astra, and generalizes to unseen objects, goals, and robot embodiments.