Representations learned via self-supervised learning (SSL) can be susceptible
to dimensional collapse, where the learned representation subspace is of
extremely low dimensionality and thus fails to represent the full data
distribution and modalities. Dimensional collapse also known as the
"underfilling" phenomenon is one of the major causes of degraded performance on
downstream tasks. Previous work has investigated the dimensional collapse
problem of SSL at a global level. In this paper, we demonstrate that
representations can span over high dimensional space globally, but collapse
locally. To address this, we propose a method called local dimensionality regularization (LDReg). Our formulation is based on the
derivation of the Fisher-Rao metric to compare and optimize local distance
distributions at an asymptotically small radius for each data point. By
increasing the local intrinsic dimensionality, we demonstrate through a range
of experiments that LDReg improves the representation quality of SSL. The
results also show that LDReg can regularize dimensionality at both local and
global levels.Comment: ICLR 202