2,584 research outputs found
Gated Class-Attention with Cascaded Feature Drift Compensation for Exemplar-free Continual Learning of Vision Transformers
In this paper we propose a new method for exemplar-free class incremental
training of ViTs. The main challenge of exemplar-free continual learning is
maintaining plasticity of the learner without causing catastrophic forgetting
of previously learned tasks. This is often achieved via exemplar replay which
can help recalibrate previous task classifiers to the feature drift which
occurs when learning new tasks. Exemplar replay, however, comes at the cost of
retaining samples from previous tasks which for some applications may not be
possible. To address the problem of continual ViT training, we first propose
gated class-attention to minimize the drift in the final ViT transformer block.
This mask-based gating is applied to class-attention mechanism of the last
transformer block and strongly regulates the weights crucial for previous
tasks. Secondly, we propose a new method of feature drift compensation that
accommodates feature drift in the backbone when learning new tasks. The
combination of gated class-attention and cascaded feature drift compensation
allows for plasticity towards new tasks while limiting forgetting of previous
ones. Extensive experiments performed on CIFAR-100, Tiny-ImageNet and
ImageNet100 demonstrate that our method outperforms existing exemplar-free
state-of-the-art methods without the need to store any representative exemplars
of past tasks
A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
Continual learning, also known as incremental learning or life-long learning,
stands at the forefront of deep learning and AI systems. It breaks through the
obstacle of one-way training on close sets and enables continuous adaptive
learning on open-set conditions. In the recent decade, continual learning has
been explored and applied in multiple fields especially in computer vision
covering classification, detection and segmentation tasks. Continual semantic
segmentation (CSS), of which the dense prediction peculiarity makes it a
challenging, intricate and burgeoning task. In this paper, we present a review
of CSS, committing to building a comprehensive survey on problem formulations,
primary challenges, universal datasets, neoteric theories and multifarious
applications. Concretely, we begin by elucidating the problem definitions and
primary challenges. Based on an in-depth investigation of relevant approaches,
we sort out and categorize current CSS models into two main branches including
\textit{data-replay} and \textit{data-free} sets. In each branch, the
corresponding approaches are similarity-based clustered and thoroughly
analyzed, following qualitative comparison and quantitative reproductions on
relevant datasets. Besides, we also introduce four CSS specialities with
diverse application scenarios and development tendencies. Furthermore, we
develop a benchmark for CSS encompassing representative references, evaluation
results and reproductions, which is available
at~\url{https://github.com/YBIO/SurveyCSS}. We hope this survey can serve as a
reference-worthy and stimulating contribution to the advancement of the
life-long learning field, while also providing valuable perspectives for
related fields.Comment: 20 pages, 12 figures. Undergoing Revie
FDCNet: Feature Drift Compensation Network for Class-Incremental Weakly Supervised Object Localization
This work addresses the task of class-incremental weakly supervised object
localization (CI-WSOL). The goal is to incrementally learn object localization
for novel classes using only image-level annotations while retaining the
ability to localize previously learned classes. This task is important because
annotating bounding boxes for every new incoming data is expensive, although
object localization is crucial in various applications. To the best of our
knowledge, we are the first to address this task. Thus, we first present a
strong baseline method for CI-WSOL by adapting the strategies of
class-incremental classifiers to mitigate catastrophic forgetting. These
strategies include applying knowledge distillation, maintaining a small data
set from previous tasks, and using cosine normalization. We then propose the
feature drift compensation network to compensate for the effects of feature
drifts on class scores and localization maps. Since updating network parameters
to learn new tasks causes feature drifts, compensating for the final outputs is
necessary. Finally, we evaluate our proposed method by conducting experiments
on two publicly available datasets (ImageNet-100 and CUB-200). The experimental
results demonstrate that the proposed method outperforms other baseline
methods.Comment: ACM Multimedia 202
ICICLE: Interpretable Class Incremental Continual Learning
Continual learning enables incremental learning of new tasks without
forgetting those previously learned, resulting in positive knowledge transfer
that can enhance performance on both new and old tasks. However, continual
learning poses new challenges for interpretability, as the rationale behind
model predictions may change over time, leading to interpretability concept
drift. We address this problem by proposing Interpretable Class-InCremental
LEarning (ICICLE), an exemplar-free approach that adopts a prototypical
part-based approach. It consists of three crucial novelties: interpretability
regularization that distills previously learned concepts while preserving
user-friendly positive reasoning; proximity-based prototype initialization
strategy dedicated to the fine-grained setting; and task-recency bias
compensation devoted to prototypical parts. Our experimental results
demonstrate that ICICLE reduces the interpretability concept drift and
outperforms the existing exemplar-free methods of common class-incremental
learning when applied to concept-based models. We make the code available.Comment: Under review, code will be shared after the acceptanc
Multivariate Prototype Representation for Domain-Generalized Incremental Learning
Deep learning models suffer from catastrophic forgetting when being
fine-tuned with samples of new classes. This issue becomes even more pronounced
when faced with the domain shift between training and testing data. In this
paper, we study the critical and less explored Domain-Generalized
Class-Incremental Learning (DGCIL). We design a DGCIL approach that remembers
old classes, adapts to new classes, and can classify reliably objects from
unseen domains. Specifically, our loss formulation maintains classification
boundaries and suppresses the domain-specific information of each class. With
no old exemplars stored, we use knowledge distillation and estimate old class
prototype drift as incremental training advances. Our prototype representations
are based on multivariate Normal distributions whose means and covariances are
constantly adapted to changing model features to represent old classes well by
adapting to the feature space drift. For old classes, we sample pseudo-features
from the adapted Normal distributions with the help of Cholesky decomposition.
In contrast to previous pseudo-feature sampling strategies that rely solely on
average mean prototypes, our method excels at capturing varying semantic
information. Experiments on several benchmarks validate our claims
Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learning
In the context of continual learning, prototypes-as representative class
embeddings-offer advantages in memory conservation and the mitigation of
catastrophic forgetting. However, challenges related to semantic drift and
prototype interference persist. In this study, we introduce the Contrastive
Prototypical Prompt (CPP) approach. Through task-specific prompt-tuning,
underpinned by a contrastive learning objective, we effectively address both
aforementioned challenges. Our evaluations on four challenging
class-incremental benchmarks reveal that CPP achieves a significant 4% to 6%
improvement over state-of-the-art methods. Importantly, CPP operates without a
rehearsal buffer and narrows the performance divergence between continual and
offline joint-learning, suggesting an innovative scheme for Transformer-based
continual learning systems.Comment: Accept to WACV 2024. Code is available at
https://github.com/LzVv123456/Contrastive-Prototypical-Promp
DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation
The Class Incremental Semantic Segmentation (CISS) extends the traditional
segmentation task by incrementally learning newly added classes. Previous work
has introduced generative replay, which involves replaying old class samples
generated from a pre-trained GAN, to address the issues of catastrophic
forgetting and privacy concerns. However, the generated images lack semantic
precision and exhibit out-of-distribution characteristics, resulting in
inaccurate masks that further degrade the segmentation performance. To tackle
these challenges, we propose DiffusePast, a novel framework featuring a
diffusion-based generative replay module that generates semantically accurate
images with more reliable masks guided by different instructions (e.g., text
prompts or edge maps). Specifically, DiffusePast introduces a dual-generator
paradigm, which focuses on generating old class images that align with the
distribution of downstream datasets while preserving the structure and layout
of the original images, enabling more precise masks. To adapt to the novel
visual concepts of newly added classes continuously, we incorporate class-wise
token embedding when updating the dual-generator. Moreover, we assign adequate
pseudo-labels of old classes to the background pixels in the new step images,
further mitigating the forgetting of previously learned knowledge. Through
comprehensive experiments, our method demonstrates competitive performance
across mainstream benchmarks, striking a better balance between the performance
of old and novel classes.Comment: e.g.: 13 pages, 7 figure
- …