11 research outputs found
Continual Contrastive Self-supervised Learning for Image Classification
For artificial learning systems, continual learning over time from a stream
of data is essential. The burgeoning studies on supervised continual learning
have achieved great progress, while the study of catastrophic forgetting in
unsupervised learning is still blank. Among unsupervised learning methods,
self-supervise learning method shows tremendous potential on visual
representation without any labeled data at scale. To improve the visual
representation of self-supervised learning, larger and more varied data is
needed. In the real world, unlabeled data is generated at all times. This
circumstance provides a huge advantage for the learning of the self-supervised
method. However, in the current paradigm, packing previous data and current
data together and training it again is a waste of time and resources. Thus, a
continual self-supervised learning method is badly needed. In this paper, we
make the first attempt to implement the continual contrastive self-supervised
learning by proposing a rehearsal method, which keeps a few exemplars from the
previous data. Instead of directly combining saved exemplars with the current
data set for training, we leverage self-supervised knowledge distillation to
transfer contrastive information among previous data to the current network by
mimicking similarity score distribution inferred by the old network over a set
of saved exemplars. Moreover, we build an extra sample queue to assist the
network to distinguish between previous and current data and prevent mutual
interference while learning their own feature representation. Experimental
results show that our method performs well on CIFAR100 and ImageNet-Sub.
Compared with the baselines, which learning tasks without taking any technique,
we improve the image classification top-1 accuracy by 1.60% on CIFAR100, 2.86%
on ImageNet-Sub and 1.29% on ImageNet-Full under 10 incremental steps setting
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning
Current deep learning research is dominated by benchmark evaluation. A method
is regarded as favorable if it empirically performs well on the dedicated test
set. This mentality is seamlessly reflected in the resurfacing area of
continual learning, where consecutively arriving sets of benchmark data are
investigated. The core challenge is framed as protecting previously acquired
representations from being catastrophically forgotten due to the iterative
parameter updates. However, comparison of individual methods is nevertheless
treated in isolation from real world application and typically judged by
monitoring accumulated test set performance. The closed world assumption
remains predominant. It is assumed that during deployment a model is guaranteed
to encounter data that stems from the same distribution as used for training.
This poses a massive challenge as neural networks are well known to provide
overconfident false predictions on unknown instances and break down in the face
of corrupted data. In this work we argue that notable lessons from open set
recognition, the identification of statistically deviating data outside of the
observed dataset, and the adjacent field of active learning, where data is
incrementally queried such that the expected performance gain is maximized, are
frequently overlooked in the deep learning era. Based on these forgotten
lessons, we propose a consolidated view to bridge continual learning, active
learning and open set recognition in deep neural networks. Our results show
that this not only benefits each individual paradigm, but highlights the
natural synergies in a common framework. We empirically demonstrate
improvements when alleviating catastrophic forgetting, querying data in active
learning, selecting task orders, while exhibiting robust open world application
where previously proposed methods fail.Comment: 32 page
Class Incremental Learning in Deep Neural Networks
With the advancement of computation capability, in particular the use of graphical processing units, deep learning systems have shown tremendous potential in achieving super-human performance in many computer vision tasks. However, deep learning models are not able to learn continuously in scenarios where the data distribution is non-stationary or imbalanced, because the models suffer from catastrophic forgetting. In this thesis, we propose an Incremental Generative Replay Embedding (IGRE) framework which employs a conditional generator for generative replay at the image embedding level, thus combining the superior performance of replay and reducing the memory complexities for replay at the same time. Alternating backpropagation with Langevin's dynamics was used for efficient and effective training of the conditional generator. We evaluate the proposed IGRE framework on common benchmarks using CIFAR10/100, CUB and ImageNet datasets. Results show that the proposed IGRE framework outperforms state-of-the-art methods on CIFAR-10, CIFAR-100, and the CUB datasets with 6-9\% improvement in accuracy and achieves comparable performance in large-scale ImageNet experiments, while at the same time reducing the memory requirements significantly when compared to conventional replay techniques