Internal variables an optimizer maintains, like momentum or adaptive learning rates, between updates.