Normalized networks follow an exact mathematical law where a single scalar quantity controls the interaction between learning rate and weight decay—and this interaction is fundamentally unstable, causing training to oscillate predictably rather than settle at a fixed point.
This paper reveals that normalization in neural networks creates a hidden feedback loop between learning rate schedules and weight decay. The authors derive an exact mathematical law governing this interaction and show it produces unstable dynamics that cause training to oscillate rather than converge smoothly.