Even though multi-step lookahead planning is computationally hard in the worst case, you can still plan near-optimally in polynomial time—and learning from this lookahead achieves the same regret bounds as standard RL without it.
This paper studies reinforcement learning where agents can preview the next ℓ states before choosing actions. While exact planning with this lookahead is NP-hard, the authors prove near-optimal planning is still possible in polynomial time using a randomized approximation scheme. They extend this to unknown environments and show the algorithm achieves regret matching standard RL methods.