Curriculum learning substantially changes language models' learning biases, suggesting that training order matters as much as model architecture when predicting which language structures are 'easy' to learn.
This paper investigates how curriculum learning—training language models on simpler sentences first rather than random order—affects which linguistic patterns models naturally learn.