Continual learning for Semantic Segmentation (CSS) is a rapidly emerging
field, in which the capabilities of the segmentation model are incrementally
improved by learning new classes or new domains. A central challenge in
Continual Learning is overcoming the effects of catastrophic forgetting, which
refers to the sudden drop in accuracy on previously learned tasks after the
model is trained on new classes or domains. In continual classification this
challenge is often overcome by replaying a small selection of samples from
previous tasks, however replay is rarely considered in CSS. Therefore, we
investigate the influences of various replay strategies for semantic
segmentation and evaluate them in class- and domain-incremental settings. Our
findings suggest that in a class-incremental setting, it is critical to achieve
a uniform distribution for the different classes in the buffer to avoid a bias
towards newly learned classes. In the domain-incremental setting, it is most
effective to select buffer samples by uniformly sampling from the distribution
of learned feature representations or by choosing samples with median entropy.
Finally, we observe that the effective sampling methods help to decrease the
representation shift significantly in early layers, which is a major cause of
forgetting in domain-incremental learning.Comment: Accepted at 2022 IEEE Conference on Intelligent Transportation
Systems (ITSC 2022