You can get ensemble-like improvements from frozen models by adding noise to predictions and voting, rather than perturbing weights—making it easier to deploy pretrained models without modification.
This paper challenges the Neural Thickets hypothesis by showing that the performance gains from perturbing pretrained model weights don't require the proposed geometric explanation. Instead, simply adding random noise to output logits and selecting/voting on predictions achieves similar or better results—without any weight changes.