Neural networks exhibit a sharp phase transition where a small fraction of trainable parameters (often <5%) can capture most of the performance gains from full training, suggesting that strategic partial fine-tuning may be as effective as training entire networks.
This paper reveals a phase transition pattern in neural networks: as you gradually make more neurons trainable (from 0% to 100%), approximation quality improves rapidly at first, then plateaus.