8 research outputs found
Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning
Intermediate features of a pre-trained model have been shown informative for
making accurate predictions on downstream tasks, even if the model backbone is
kept frozen. The key challenge is how to utilize these intermediate features
given their gigantic amount. We propose visual query tuning (VQT), a simple yet
effective approach to aggregate intermediate features of Vision Transformers.
Through introducing a handful of learnable ``query'' tokens to each layer, VQT
leverages the inner workings of Transformers to ``summarize'' rich intermediate
features of each layer, which can then be used to train the prediction heads of
downstream tasks. As VQT keeps the intermediate features intact and only learns
to combine them, it enjoys memory efficiency in training, compared to many
other parameter-efficient fine-tuning approaches that learn to adapt features
and need back-propagation through the entire backbone. This also suggests the
complementary role between VQT and those approaches in transfer learning.
Empirically, VQT consistently surpasses the state-of-the-art approach that
utilizes intermediate features for transfer learning and outperforms full
fine-tuning in many cases. Compared to parameter-efficient approaches that
adapt features, VQT achieves much higher accuracy under memory constraints.
Most importantly, VQT is compatible with these approaches to attain even higher
accuracy, making it a simple add-on to further boost transfer learning.Comment: Accepted by CVPR 2023. Cheng-Hao Tu and Zheda Mai contributed equally
to this wor
Online Class-Incremental Continual Learning with Adversarial Shapley Value
As image-based deep learning becomes pervasive on every device, from cell
phones to smart watches, there is a growing need to develop methods that
continually learn from data while minimizing memory footprint and power
consumption. While memory replay techniques have shown exceptional promise for
this task of continual learning, the best method for selecting which buffered
images to replay is still an open question. In this paper, we specifically
focus on the online class-incremental setting where a model needs to learn new
classes continually from an online data stream. To this end, we contribute a
novel Adversarial Shapley value scoring method that scores memory data samples
according to their ability to preserve latent decision boundaries for
previously observed classes (to maintain learning stability and avoid
forgetting) while interfering with latent decision boundaries of current
classes being learned (to encourage plasticity and optimal learning of new
class boundaries). Overall, we observe that our proposed ASER method provides
competitive or improved performance compared to state-of-the-art replay-based
continual learning methods on a variety of datasets.Comment: Proceedings of the 35th AAAI Conference on Artificial Intelligence
(AAAI-21
Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data
We propose a learning problem involving adapting a pre-trained source model
to the target domain for classifying all classes that appeared in the source
data, using target data that covers only a partial label space. This problem is
practical, as it is unrealistic for the target end-users to collect data for
all classes prior to adaptation. However, it has received limited attention in
the literature. To shed light on this issue, we construct benchmark datasets
and conduct extensive experiments to uncover the inherent challenges. We found
a dilemma -- on the one hand, adapting to the new target domain is important to
claim better performance; on the other hand, we observe that preserving the
classification accuracy of classes missing in the target adaptation data is
highly challenging, let alone improving them. To tackle this, we identify two
key directions: 1) disentangling domain gradients from classification
gradients, and 2) preserving class relationships. We present several effective
solutions that maintain the accuracy of the missing classes and enhance the
overall performance, establishing solid baselines for holistic transfer of
pre-trained models with partial target data.Comment: Accepted to NeurIPS 2023 main trac
Online Continual Learning in Image Classification
Online continual learning (CL) in image classification studies the problem of learning to classify images from an online stream of data and tasks, where tasks may include data with new classes or nonstationarity. This thesis starts with a comparative empirical survey of the state-of-the-art methods and recently proposed tricks to better understand their relative advantages and the settings where they work the best. To prepare CL for real-world applications, we provide a simple but effective approach that combines memory replay, CL tricks summarized in the survey and common image classification techniques. Moreover, for replay-based approaches, we propose a novel Adversarial Shapley value scoring method for memory sample selection that scores samples according to their abilities to preserve latent decision boundaries for previously observed classes while interfering with latent decision boundaries of current classes being learned. Our proposed method shows state-of-the-art performance, especially when the memory buffer size is small.M.A.S
Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation
Weakly Supervised Semantic Segmentation (WSSS) with only image-level
supervision has garnered increasing attention due to its low annotation cost
compared to pixel-level annotation. Most existing methods rely on Class
Activation Maps (CAM) to generate pixel-level pseudo labels for supervised
training. However, it is well known that CAM often suffers from partial
activation -- activating the most discriminative part instead of the entire
object area, and false activation -- unnecessarily activating the background
around the object. In this study, we introduce a simple yet effective approach
to address these limitations by harnessing the recently released Segment
Anything Model (SAM) to generate higher-quality pseudo labels with CAM. SAM is
a segmentation foundation model that demonstrates strong zero-shot ability in
partitioning images into segments but lacks semantic labels for these regions.
To circumvent this, we employ pseudo labels for a specific class as the signal
to select the most relevant masks and label them to generate the refined pseudo
labels for this class. The segments generated by SAM are highly precise,
leading to substantial improvements in partial and false activation. Moreover,
existing post-processing modules for producing pseudo labels, such as
AffinityNet, are often computationally heavy, with a significantly long
training time. Surprisingly, we discovered that using the initial CAM with SAM
can achieve on-par performance as the post-processed pseudo label generated
from these modules with much less computational cost. Our approach is highly
versatile and capable of seamless integration into existing WSSS models without
modification to base networks or pipelines. Despite its simplicity, our
approach improves the mean Intersection over Union (mIoU) of pseudo labels from
five state-of-the-art WSSS methods by 6.2\% on average on the PASCAL VOC 2012
dataset.Comment: Tianle Chen and Zheda Mai contributed equally to this work. Our code
is available at \url{https://github.com/cskyl/SAM_WSSS