Pretrained video generators can serve as powerful priors for discovering latent actions from visual transitions, enabling few-shot and zero-shot action transfer across different visual contexts and robot embodiments.
This paper proposes O²LaF, a method for learning actions from unlabeled videos by repurposing pretrained video diffusion transformers. The approach uses a single pretrained model to initialize both inverse dynamics (what action caused this change?) and forward dynamics (what happens after this action?), learning transferable actions without manual labels.