Tactile sensing matters for dexterous manipulation: modeling contact evolution as part of world state (not just as input) nearly triples performance, and pretrained vision models can efficiently extend to touch with minimal data.
DexTacWAM combines vision and touch sensing for robot hand manipulation by extending video prediction models with tactile data from fingertips. The system predicts both visual and contact dynamics together, achieving 70% success on complex multi-finger tasks—nearly double the best existing approach—while learning from minimal touch data by adapting a pretrained vision model.