Memory sufficient for making accurate predictions is not sufficient for learning from new data—you need extra hidden state to capture what future observations will teach you, and the amount depends on your loss function's degree.
This paper proves fundamental limits on memory requirements for learning systems. It shows that a recurrent state reproducing current predictions may still lack information needed to learn from future observations.