What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja et al.|April 9, 2026arXiv

Key Takeaway

Language models don't learn randomly during pretraining; they follow a structured, predictable curriculum where complex skills emerge from simpler ones in a consistent order across models, which you can predict from analyzing the model's internal representations.

Summary

This paper reveals that language models learn skills in a predictable, compositional order during training—like following a hidden curriculum.

training reasoning evaluation

Key Terms

emergence compositional-generalization internal-representations pretraining