LeCun Continuously Endorses, New Work VISReg Tackles the Core Challenge of 'Representation Collapse' in JEPA World Models
"VISReg: A New Self-Supervised Learning Method Tackles Representation Collapse in JEPA World Models"
Yann LeCun has highlighted the new self-supervised learning (SSL) work VISReg (Variance-Invariance-Sketching Regularization), which addresses the core challenge of "representation collapse" in JEPA-based world models. SSL often collapses, mapping different inputs to similar vectors, losing discriminative power. While methods like VICReg and SIGReg attempted to regularize the representation distribution, they suffered from issues like vanishing gradients during collapse or coupled scale and shape optimization.
VISReg overcomes these by decoupling the regularization into independent "scale" and "shape" objectives. It uses a variance term to prevent amplitude collapse (providing constant gradient even during collapse) and a sliced Wasserstein distance (SWD) based sketching target with stop-gradient to align the distribution shape to an isotropic Gaussian, without interfering with scale. This approach, requiring only ~15 lines of PyTorch core code, offers linear computational complexity and scales efficiently across multiple GPUs.
Evaluated across 15 datasets (in-domain, out-of-distribution/OOD, dense prediction), VISReg outperforms 7 mainstream SSL methods (MoCoV3, DINO, iBOT, I-JEPA, MAE, data2vec) without relying on heuristic tricks like EMA or stop-gradient. Key results include: superior average OOD accuracy; matching DINOv2's OOD performance using only ~1/10 of the data (ImageNet-22K vs. LVD-142M); better transfer learning fine-tuning results across multiple datasets; and competitive performance in semantic segmentation and generative guidance. VISReg also demonstrates robustness on low-quality data like long-tailed or low-rank datasets where other methods fail. The work provides a more stable, efficient, and generalizable regularization solution for SSL and JEPA models.
marsbit8 h fa
