We introduce a novel and general loss function, called Symmetric Contrastive
(Sy-CON) loss, for effective continual self-supervised learning (CSSL). We
first argue that the conventional loss form of continual learning which
consists of single task-specific loss (for plasticity) and a regularizer (for
stability) may not be ideal for contrastive loss based CSSL that focus on
representation learning. Our reasoning is that, in contrastive learning based
methods, the task-specific loss would suffer from decreasing diversity of
negative samples and the regularizer may hinder learning new distinctive
representations. To that end, we propose Sy-CON that consists of two losses
(one for plasticity and the other for stability) with symmetric dependence on
current and past models' negative sample embeddings. We argue our model can
naturally find good trade-off between the plasticity and stability without any
explicit hyperparameter tuning. We validate the effectiveness of our approach
through extensive experiments, demonstrating that MoCo-based implementation of
Sy-CON loss achieves superior performance compared to other state-of-the-art
CSSL methods.Comment: Preprin