Due to the limited availability of data, existing few-shot learning methods
trained from scratch fail to achieve satisfactory performance. In contrast,
large-scale pre-trained models such as CLIP demonstrate remarkable few-shot and
zero-shot capabilities. To enhance the performance of pre-trained models for
downstream tasks, fine-tuning the model on downstream data is frequently
necessary. However, fine-tuning the pre-trained model leads to a decrease in
its generalizability in the presence of distribution shift, while the limited
number of samples in few-shot learning makes the model highly susceptible to
overfitting. Consequently, existing methods for fine-tuning few-shot learning
primarily focus on fine-tuning the model's classification head or introducing
additional structure. In this paper, we introduce a fine-tuning approach termed
Feature Discrimination Alignment (FD-Align). Our method aims to bolster the
model's generalizability by preserving the consistency of spurious features
across the fine-tuning process. Extensive experimental results validate the
efficacy of our approach for both ID and OOD tasks. Once fine-tuned, the model
can seamlessly integrate with existing methods, leading to performance
improvements. Our code can be found in https://github.com/skingorz/FD-Align.Comment: Accepted by NeurIPS 202