Visual Contrastive Self-Distillation

Yijun Liang, Yunjie Tian, Yijiang Li, Yuqi Jia, Furong Huang et al.|July 23, 2026arXiv

Key Takeaway

VCSD achieves significant improvements in vision-language model performance (up to 5% on benchmarks) by using contrastive image conditioning during self-distillation, requiring no external teachers, privileged data, or extra inference costs.

Summary

This paper introduces Visual Contrastive Self-Distillation (VCSD), a training method that improves vision-language models by having them learn from themselves without needing an external teacher.

training multimodal efficiency

Key Terms

self-distillation on-policy-distillation ema-teacher contrastive-learning vision-language-models