Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student model based on the knowledge from a teacher model. However, there have been very few efforts for applying KD in image regression with a scalar response, and there is no KD method applicable to both tasks. Moreover, existing KD methods often require a practitioner to carefully choose or adjust the teacher and student architectures, making these methods less scalable in practice. Furthermore, although KD is usually conducted in scenarios with limited labeled data, very few techniques are developed to alleviate such data insufficiency. To solve the above problems in an all-in-one manner, we propose in this paper a unified KD framework based on conditional generative adversarial networks (cGANs), termed cGAN-KD. Fundamentally different from existing KD methods, cGAN-KD distills and transfers knowledge from a teacher model to a student model via cGAN-generated samples. This unique mechanism makes cGAN-KD suitable for both classification and regression tasks, compatible with other KD methods, and insensitive to the teacher and student architectures. Also, benefiting from the recent advances in cGAN methodology and our specially designed subsampling and filtering procedures, cGAN-KD also performs well when labeled data are scarce. An error bound of a student model trained in the cGAN-KD framework is derived in this work, which theoretically explains why cGAN-KD takes effect and guides the implementation of cGAN-KD in practice. Extensive experiments on CIFAR-10 and Tiny-ImageNet show that we can incorporate state-of-the-art KD methods into the cGAN-KD framework to reach a new state of the art. Also, experiments on RC-49 and UTKFace demonstrate the effectiveness of cGAN-KD in image regression tasks, where existing KD methods are inapplicable

ECG Arrhythmia Classification Using Transfer Learning from 2-Dimensional Deep CNN Features

Due to the recent advances in the area of deep learning, it has been demonstrated that a deep neural network, trained on a huge amount of data, can recognize cardiac arrhythmias better than cardiologists. Moreover, traditionally feature extraction was considered an integral part of ECG pattern recognition; however, recent findings have shown that deep neural networks can carry out the task of feature extraction directly from the data itself. In order to use deep neural networks for their accuracy and feature extraction, high volume of training data is required, which in the case of independent studies is not pragmatic. To arise to this challenge, in this work, the identification and classification of four ECG patterns are studied from a transfer learning perspective, transferring knowledge learned from the image classification domain to the ECG signal classification domain. It is demonstrated that feature maps learned in a deep neural network trained on great amounts of generic input images can be used as general descriptors for the ECG signal spectrograms and result in features that enable classification of arrhythmias. Overall, an accuracy of 97.23 percent is achieved in classifying near 7000 instances by ten-fold cross validation.Comment: Accepted and presented for IEEE Biomedical Circuits and Systems (BioCAS) on 17th-19th October 2018 in Ohio, US

Knowledge Transfer via Multi-Head Feature Adaptation for Whole Slide Image Classification

Transferring prior knowledge from a source domain to the same or similar target domain can greatly enhance the performance of models on the target domain. However, it is challenging to directly leverage the knowledge from the source domain due to task discrepancy and domain shift. To bridge the gaps between different tasks and domains, we propose a Multi-Head Feature Adaptation module, which projects features in the source feature space to a new space that is more similar to the target space. Knowledge transfer is particularly important in Whole Slide Image (WSI) classification since the number of WSIs in one dataset might be too small to achieve satisfactory performance. Therefore, WSI classification is an ideal testbed for our method, and we adapt multiple knowledge transfer methods for WSI classification. The experimental results show that models with knowledge transfer outperform models that are trained from scratch by a large margin regardless of the number of WSIs in the datasets, and our method achieves state-of-the-art performances among other knowledge transfer methods on multiple datasets, including TCGA-RCC, TCGA-NSCLC, and Camelyon16 datasets