Contrastive learning (CL) methods effectively learn data representations in a
self-supervision manner, where the encoder contrasts each positive sample over
multiple negative samples via a one-vs-many softmax cross-entropy loss. By
leveraging large amounts of unlabeled image data, recent CL methods have
achieved promising results when pretrained on large-scale datasets, such as
ImageNet. However, most of them consider the augmented views from the same
instance are positive pairs, while views from other instances are negative
ones. Such binary partition insufficiently considers the relation between
samples and tends to yield worse performance when generalized on images in the
wild. In this paper, to further improve the performance of CL and enhance its
robustness on various datasets, {we propose a doubly CL strategy that
separately compares positive and negative samples within their own groups, and
then proceeds with a contrast between positive and negative groups}. We realize
this strategy with contrastive attraction and contrastive repulsion (CACR),
which makes the query not only exert a greater force to attract more distant
positive samples but also do so to repel closer negative samples. Theoretical
analysis reveals that CACR generalizes CL's behavior by positive attraction and
negative repulsion, and it further considers the intra-contrastive relation
within the positive and negative pairs to narrow the gap between the sampled
and true distribution, which is important when datasets are less curated. With
our extensive experiments, CACR not only demonstrates good performance on CL
benchmarks, but also shows better robustness when generalized on imbalanced
image datasets. Code and pre-trained checkpoints are available at
https://github.com/JegZheng/CACR-SSL