Autoregressive pretraining on video unlocks the value of long context windows for robot control—bidirectional models don't benefit from extra history, but AR-pretrained models show consistent improvements with longer visual context.
Long-WAM is a system for controlling robots in real-time by processing long video histories (up to 19 seconds) to understand motion and task progress. The key insight is that autoregressive video pretraining—learning to predict future frames from past ones—makes longer context windows actually useful for robot control, whereas bidirectional pretraining doesn't benefit from extra history.