Data quality trumps quantity — a tiny model trained on curated textbook-quality data can outperform much larger models trained on web scrapes.
Demonstrates that a 1.3B parameter model (Phi-1) trained on high-quality textbook data outperforms models 10x its size on code generation. Data quality matters more than data quantity for small models.