You do not need to update all parameters to adapt a large model — low-rank updates in attention layers are sufficient, making fine-tuning accessible on consumer hardware.
Proposes Low-Rank Adaptation, which freezes pre-trained weights and injects trainable low-rank matrices into transformer layers. Reduces trainable parameters by 10,000x while matching full fine-tuning quality.