20 research outputs found
Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution
Knowledge distillation (KD) is an effective framework that aims to transfer
meaningful information from a large teacher to a smaller student. Generally, KD
often involves how to define and transfer knowledge. Previous KD methods often
focus on mining various forms of knowledge, for example, feature maps and
refined information. However, the knowledge is derived from the primary
supervised task and thus is highly task-specific. Motivated by the recent
success of self-supervised representation learning, we propose an auxiliary
self-supervision augmented task to guide networks to learn more meaningful
features. Therefore, we can derive soft self-supervision augmented
distributions as richer dark knowledge from this task for KD. Unlike previous
knowledge, this distribution encodes joint knowledge from supervised and
self-supervised feature learning. Beyond knowledge exploration, we propose to
append several auxiliary branches at various hidden layers, to fully take
advantage of hierarchical feature maps. Each auxiliary branch is guided to
learn self-supervision augmented task and distill this distribution from
teacher to student. Overall, we call our KD method as Hierarchical
Self-Supervision Augmented Knowledge Distillation (HSSAKD). Experiments on
standard image classification show that both offline and online HSSAKD achieves
state-of-the-art performance in the field of KD. Further transfer experiments
on object detection further verify that HSSAKD can guide the network to learn
better features. The code is available at https://github.com/winycg/HSAKD.Comment: 15 pages, Accepted by IEEE Transactions on Neural Networks and
Learning Systems 202
CLIP-KD: An Empirical Study of Distilling CLIP Models
CLIP has become a promising language-supervised visual pre-training framework
and achieves excellent performance over a wide range of tasks. This paper aims
to distill small CLIP models supervised by a large teacher CLIP model. We
propose several distillation strategies, including relation, feature, gradient
and contrastive paradigm, to examine the impact on CLIP distillation. We show
that the simplest feature mimicry with MSE loss performs best. Moreover,
interactive contrastive learning and relation-based distillation are also
critical in performance improvement. We apply the unified method to distill
several student networks trained on 15 million (image, text) pairs.
Distillation improves the student CLIP models consistently over zero-shot
ImageNet classification and cross-modal retrieval benchmarks. We hope our
empirical study will become an important baseline for future CLIP distillation
research. The code is available at \url{https://github.com/winycg/CLIP-KD}
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
The teacher-free online Knowledge Distillation (KD) aims to train an ensemble
of multiple student models collaboratively and distill knowledge from each
other. Although existing online KD methods achieve desirable performance, they
often focus on class probabilities as the core knowledge type, ignoring the
valuable feature representational information. We present a Mutual Contrastive
Learning (MCL) framework for online KD. The core idea of MCL is to perform
mutual interaction and transfer of contrastive distributions among a cohort of
networks in an online manner. Our MCL can aggregate cross-network embedding
information and maximize the lower bound to the mutual information between two
networks. This enables each network to learn extra contrastive knowledge from
others, leading to better feature representations, thus improving the
performance of visual recognition tasks. Beyond the final layer, we extend MCL
to intermediate layers and perform an adaptive layer-matching mechanism trained
by meta-optimization. Experiments on image classification and transfer learning
to visual recognition tasks show that layer-wise MCL can lead to consistent
performance gains against state-of-the-art online KD approaches. The
superiority demonstrates that layer-wise MCL can guide the network to generate
better feature representations. Our code is publicly avaliable at
https://github.com/winycg/L-MCL.Comment: 18 pages, accepted by IEEE Transactions on Pattern Analysis and
Machine Intelligence (TPAMI-2023
Localizing Semantic Patches for Accelerating Image Classification
Existing works often focus on reducing the architecture redundancy for
accelerating image classification but ignore the spatial redundancy of the
input image. This paper proposes an efficient image classification pipeline to
solve this problem. We first pinpoint task-aware regions over the input image
by a lightweight patch proposal network called AnchorNet. We then feed these
localized semantic patches with much smaller spatial redundancy into a general
classification network. Unlike the popular design of deep CNN, we aim to
carefully design the Receptive Field of AnchorNet without intermediate
convolutional paddings. This ensures the exact mapping from a high-level
spatial location to the specific input image patch. The contribution of each
patch is interpretable. Moreover, AnchorNet is compatible with any downstream
architecture. Experimental results on ImageNet show that our method outperforms
SOTA dynamic inference methods with fewer inference costs. Our code is
available at https://github.com/winycg/AnchorNet.Comment: Accepted by ICME-202