By interpolating between old and new model checkpoints with learned per-parameter scalars, you can adapt multimodal models to new data while preserving earlier knowledge—all without changing model size or inference speed.
AlphaWiSE solves a key problem in continual learning for multimodal models like CLIP: when models adapt to new data sequentially, they forget what they learned before. This method blends two frozen model checkpoints using learned scalar weights per parameter, maintaining cross-modal alignment without extra computation cost.