Improving Replay-Based Continual Semantic Segmentation with Smart Data Selection

Continual learning for Semantic Segmentation (CSS) is a rapidly emerging field, in which the capabilities of the segmentation model are incrementally improved by learning new classes or new domains. A central challenge in Continual Learning is overcoming the effects of catastrophic forgetting, which refers to the sudden drop in accuracy on previously learned tasks after the model is trained on new classes or domains. In continual classification this challenge is often overcome by replaying a small selection of samples from previous tasks, however replay is rarely considered in CSS. Therefore, we investigate the influences of various replay strategies for semantic segmentation and evaluate them in class- and domain-incremental settings. Our findings suggest that in a class-incremental setting, it is critical to achieve a uniform distribution for the different classes in the buffer to avoid a bias towards newly learned classes. In the domain-incremental setting, it is most effective to select buffer samples by uniformly sampling from the distribution of learned feature representations or by choosing samples with median entropy. Finally, we observe that the effective sampling methods help to decrease the representation shift significantly in early layers, which is a major cause of forgetting in domain-incremental learning.Comment: Accepted at 2022 IEEE Conference on Intelligent Transportation Systems (ITSC 2022

Balanced Softmax Cross-Entropy for Incremental Learning

Deep neural networks are prone to catastrophic forgetting when incrementally trained on new classes or new tasks as adaptation to the new data leads to a drastic decrease of the performance on the old classes and tasks. By using a small memory for rehearsal and knowledge distillation, recent methods have proven to be effective to mitigate catastrophic forgetting. However due to the limited size of the memory, large imbalance between the amount of data available for the old and new classes still remains which results in a deterioration of the overall accuracy of the model. To address this problem, we propose the use of the Balanced Softmax Cross-Entropy loss and show that it can be combined with exiting methods for incremental learning to improve their performances while also decreasing the computational cost of the training procedure in some cases. Complete experiments on the competitive ImageNet, subImageNet and CIFAR100 datasets show states-of-the-art results

Hypothesis-driven Online Video Stream Learning with Augmented Memory

The ability to continuously acquire new knowledge without forgetting previous tasks remains a challenging problem for computer vision systems. Standard continual learning benchmarks focus on learning from static iid images in an offline setting. Here, we examine a more challenging and realistic online continual learning problem called online stream learning. Like humans, some AI agents have to learn incrementally from a continuous temporal stream of non-repeating data. We propose a novel model, Hypotheses-driven Augmented Memory Network (HAMN), which efficiently consolidates previous knowledge using an augmented memory matrix of "hypotheses" and replays reconstructed image features to avoid catastrophic forgetting. Compared with pixel-level and generative replay approaches, the advantages of HAMN are two-fold. First, hypothesis-based knowledge consolidation avoids redundant information in the image pixel space and makes memory usage far more efficient. Second, hypotheses in the augmented memory can be re-used for learning new tasks, improving generalization and transfer learning ability. Given a lack of online incremental class learning datasets on video streams, we introduce and adapt two additional video datasets, Toybox and iLab, for online stream learning. We also evaluate our method on the CORe50 and online CIFAR100 datasets. Our method performs significantly better than all state-of-the-art methods, while offering much more efficient memory usage. All source code and data are publicly available at https://github.com/kreimanlab/AugMe

How do Human Users Teach a Continual Learning Robot in Repeated Interactions?

© IEEE 2023 ALL RIGHT RESERVED