Adding recurrence to just 20% of a transformer's middle layers—rather than all layers—can improve reasoning ability with minimal inference cost, and this works even when retrofitted onto already-trained models.
This paper introduces T²MLR, a technique that adds recurrent connections to transformer models by caching middle-layer representations and feeding them back into earlier layers during decoding.