A fact's usefulness for learning new tasks depends on whether its structure is linearly readable in the pretrained model, not just whether the model can recall it. This suggests fine-tuning success is determined by the geometry of learned representations, not memorization.
This paper shows that language models generalize better when fine-tuning on new tasks if the underlying knowledge was already linearly readable in the pretrained model.