VISReg: Variance-Invariance-Sketching Regularization for JEPA Training

Researchers introduced VISReg, a self-supervised regularizer for JEPA that replaces covariance penalties with a Sliced-Wasserstein sketching objective while preserving variance control.

The method is designed to reduce representation collapse, provide stronger gradients, and better capture full distribution structure during training.

Experiments show better scaling and improved robustness on low-quality, long-tailed, and low-rank data.

VISReg achieves state-of-the-art OOD performance after ImageNet-1K pretraining and matches DINOv2 on ImageNet-22K with far less data.