Incremental semantic segmentation aims to continually learn the segmentation
of new coming classes without accessing the training data of previously learned
classes. However, most current methods fail to address catastrophic forgetting
and background shift since they 1) treat all previous classes equally without
considering different forgetting paces caused by imbalanced gradient
back-propagation; 2) lack strong semantic guidance between classes. To tackle
the above challenges, in this paper, we propose a Gradient-Semantic
Compensation (GSC) model, which surmounts incremental semantic segmentation
from both gradient and semantic perspectives. Specifically, to address
catastrophic forgetting from the gradient aspect, we develop a step-aware
gradient compensation that can balance forgetting paces of previously seen
classes via re-weighting gradient backpropagation. Meanwhile, we propose a
soft-sharp semantic relation distillation to distill consistent inter-class
semantic relations via soft labels for alleviating catastrophic forgetting from
the semantic aspect. In addition, we develop a prototypical pseudo re-labeling
that provides strong semantic guidance to mitigate background shift. It
produces high-quality pseudo labels for old classes in the background by
measuring distances between pixels and class-wise prototypes. Extensive
experiments on three public datasets, i.e., Pascal VOC 2012, ADE20K, and
Cityscapes, demonstrate the effectiveness of our proposed GSC model