Supervising observation tokens during SFT alongside action tokens improves downstream RL performance by preventing the policy from specializing away from environment modeling, enabling better exploration without extra data or parameters.
This paper shows that training language model agents to predict both actions AND environment observations (not just actions) leads to better exploration during reinforcement learning. Even though deployed agents never generate observations, learning to predict them helps the policy understand action consequences, resulting in higher pass rates on coding tasks without adding computational overhead.