A desirable objective in self-supervised learning (SSL) is to avoid feature
collapse. Whitening loss guarantees collapse avoidance by minimizing the
distance between embeddings of positive pairs under the conditioning that the
embeddings from different views are whitened. In this paper, we propose a
framework with an informative indicator to analyze whitening loss, which
provides a clue to demystify several interesting phenomena as well as a
pivoting point connecting to other SSL methods. We reveal that batch whitening
(BW) based methods do not impose whitening constraints on the embedding, but
they only require the embedding to be full-rank. This full-rank constraint is
also sufficient to avoid dimensional collapse. Based on our analysis, we
propose channel whitening with random group partition (CW-RGP), which exploits
the advantages of BW-based methods in preventing collapse and avoids their
disadvantages requiring large batch size. Experimental results on ImageNet
classification and COCO object detection reveal that the proposed CW-RGP
possesses a promising potential for learning good representations. The code is
available at https://github.com/winci-ai/CW-RGP.Comment: Accepted at NeurIPS 2022. The Code is available at:
https://github.com/winci-ai/CW-RG