Individual training examples can be learned and then completely forgotten during pre-training, leaving virtually no measurable impact on final model behavior or internal structure, suggesting that what matters for final performance is the aggregate signal, not individual data points.
Researchers trained 32 GPT-2 models from scratch and injected a single training example at peak learning rate to measure its actual impact. The example was learned immediately but completely forgotten by the end of training, leaving no detectable trace in the final model's weights, geometry, or performance—despite moving the model within its loss basin during training.