LLMs learn knowledge more effectively when exposed to multiple reformulations of the same concept rather than just repeated text, suggesting that data diversity—not just quantity—is fundamental to pre-training success.
This paper investigates how LLMs learn knowledge during pre-training by studying the role of 'auxiliary views'—different reformulations of the same knowledge. Through controlled experiments, the researchers show that diverse representations of knowledge improve learning better than simple repetition, even when the total number of tokens stays the same.