Just because a parameter is important for inference doesn't mean training it in isolation will work—effective fine-tuning needs structured updates across entire layers, not surgical targeting of individual weights.
This paper challenges the assumption that 'Super Weights'—individual parameters whose removal severely hurts model performance—are good targets for selective training. The authors show that training only these supposedly critical parameters actually fails catastrophically, while training random parameters in the same layers works fine.