Transfer learning is a promising technique for medical image classification,
particularly for long-tailed datasets. However, the scarcity of data in medical
imaging domains often leads to overparameterization when fine-tuning large
publicly available pre-trained models. Moreover, these large models are
ineffective in deployment in clinical settings due to their computational
expenses. To address these challenges, we propose FoPro-KD, a novel approach
that unleashes the power of frequency patterns learned from frozen publicly
available pre-trained models to enhance their transferability and compression.
FoPro-KD comprises three modules: Fourier prompt generator (FPG), effective
knowledge distillation (EKD), and adversarial knowledge distillation (AKD). The
FPG module learns to generate targeted perturbations conditional on a target
dataset, exploring the representations of a frozen pre-trained model, trained
on natural images. The EKD module exploits these generalizable representations
through distillation to a smaller target model, while the AKD module further
enhances the distillation process. Through these modules, FoPro-KD achieves
significant improvements in performance on long-tailed medical image
classification benchmarks, demonstrating the potential of leveraging the
learned frequency patterns from pre-trained models to enhance transfer learning
and compression of large pre-trained models for feasible deployment