Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu et al.|September 21, 2026arXiv

Key Takeaway

To improve multi-turn tool use, don't train on all failures equally—use diagnostic methods to identify which specific model calls actually control task success, then focus training there.

Summary

This paper solves a key problem in multi-turn tool use: identifying which model calls are worth training on. When a task fails, multiple calls could be responsible, but reward signals get muddied by randomness in later steps. The method diagnoses which calls actually control success by separating action-dependent reward changes from downstream noise, then trains only those critical calls.

agents reasoning

Key Terms

multi-turn-interaction contextual-bandit-training nested-sampling tool-use reference-policy