Knowledge that appears inconsistently across web snapshots is learned faster during later pretraining stages and achieves better performance, suggesting that temporal variability in training data is a useful signal for understanding and predicting learning dynamics.
This paper investigates how language models learn different types of knowledge during pretraining by analyzing models trained on web data from different years.