You can prune any pre-trained transformer to run at variable speeds without retraining by jointly ranking all neurons and heads on a common scale—enabling one model to adapt to different hardware and latency constraints.
This paper introduces a method to make pre-trained transformers 'elastic'—able to dynamically adjust their computational cost on-the-fly without retraining. By jointly ranking individual neurons and attention heads across the entire model and optimizing under soft pruning at multiple sparsity levels simultaneously, the approach enables smooth performance degradation from full to 60% sparsity.