11,810 research outputs found
A Simple and Generic Framework for Feature Distillation via Channel-wise Transformation
Knowledge distillation is a popular technique for transferring the knowledge
from a large teacher model to a smaller student model by mimicking. However,
distillation by directly aligning the feature maps between teacher and student
may enforce overly strict constraints on the student thus degrade the
performance of the student model. To alleviate the above feature misalignment
issue, existing works mainly focus on spatially aligning the feature maps of
the teacher and the student, with pixel-wise transformation. In this paper, we
newly find that aligning the feature maps between teacher and student along the
channel-wise dimension is also effective for addressing the feature
misalignment issue. Specifically, we propose a learnable nonlinear channel-wise
transformation to align the features of the student and the teacher model.
Based on it, we further propose a simple and generic framework for feature
distillation, with only one hyper-parameter to balance the distillation loss
and the task specific loss. Extensive experimental results show that our method
achieves significant performance improvements in various computer vision tasks
including image classification (+3.28% top-1 accuracy for MobileNetV1 on
ImageNet-1K), object detection (+3.9% bbox mAP for ResNet50-based Faster-RCNN
on MS COCO), instance segmentation (+2.8% Mask mAP for ResNet50-based
Mask-RCNN), and semantic segmentation (+4.66% mIoU for ResNet18-based PSPNet in
semantic segmentation on Cityscapes), which demonstrates the effectiveness and
the versatility of the proposed method. The code will be made publicly
available.Comment: 13 page
Take a Prior from Other Tasks for Severe Blur Removal
Recovering clear structures from severely blurry inputs is a challenging
problem due to the large movements between the camera and the scene. Although
some works apply segmentation maps on human face images for deblurring, they
cannot handle natural scenes because objects and degradation are more complex,
and inaccurate segmentation maps lead to a loss of details. For general scene
deblurring, the feature space of the blurry image and corresponding sharp image
under the high-level vision task is closer, which inspires us to rely on other
tasks (e.g. classification) to learn a comprehensive prior in severe blur
removal cases. We propose a cross-level feature learning strategy based on
knowledge distillation to learn the priors, which include global contexts and
sharp local structures for recovering potential details. In addition, we
propose a semantic prior embedding layer with multi-level aggregation and
semantic attention transformation to integrate the priors effectively. We
introduce the proposed priors to various models, including the UNet and other
mainstream deblurring baselines, leading to better performance on severe blur
removal. Extensive experiments on natural image deblurring benchmarks and
real-world images, such as GoPro and RealBlur datasets, demonstrate our
method's effectiveness and generalization ability
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
Multi-label image classification is a fundamental but challenging task
towards general visual understanding. Existing methods found the region-level
cues (e.g., features from RoIs) can facilitate multi-label classification.
Nevertheless, such methods usually require laborious object-level annotations
(i.e., object labels and bounding boxes) for effective learning of the
object-level visual features. In this paper, we propose a novel and efficient
deep framework to boost multi-label classification by distilling knowledge from
weakly-supervised detection task without bounding box annotations.
Specifically, given the image-level annotations, (1) we first develop a
weakly-supervised detection (WSD) model, and then (2) construct an end-to-end
multi-label image classification framework augmented by a knowledge
distillation module that guides the classification model by the WSD model
according to the class-level predictions for the whole image and the
object-level visual features for object RoIs. The WSD model is the teacher
model and the classification model is the student model. After this cross-task
knowledge distillation, the performance of the classification model is
significantly improved and the efficiency is maintained since the WSD model can
be safely discarded in the test phase. Extensive experiments on two large-scale
datasets (MS-COCO and NUS-WIDE) show that our framework achieves superior
performances over the state-of-the-art methods on both performance and
efficiency.Comment: accepted by ACM Multimedia 2018, 9 pages, 4 figures, 5 table
ThumbNet: One Thumbnail Image Contains All You Need for Recognition
Although deep convolutional neural networks (CNNs) have achieved great
success in computer vision tasks, its real-world application is still impeded
by its voracious demand of computational resources. Current works mostly seek
to compress the network by reducing its parameters or parameter-incurred
computation, neglecting the influence of the input image on the system
complexity. Based on the fact that input images of a CNN contain substantial
redundancy, in this paper, we propose a unified framework, dubbed as ThumbNet,
to simultaneously accelerate and compress CNN models by enabling them to infer
on one thumbnail image. We provide three effective strategies to train
ThumbNet. In doing so, ThumbNet learns an inference network that performs
equally well on small images as the original-input network on large images.
With ThumbNet, not only do we obtain the thumbnail-input inference network that
can drastically reduce computation and memory requirements, but also we obtain
an image downscaler that can generate thumbnail images for generic
classification tasks. Extensive experiments show the effectiveness of ThumbNet,
and demonstrate that the thumbnail-input inference network learned by ThumbNet
can adequately retain the accuracy of the original-input network even when the
input images are downscaled 16 times
Efficient Ultrasound Image Analysis Models with Sonographer Gaze Assisted Distillation.
Recent automated medical image analysis methods have attained state-of-the-art performance but have relied on memory and compute-intensive deep learning models. Reducing model size without significant loss in performance metrics is crucial for time and memory-efficient automated image-based decision-making. Traditional deep learning based image analysis only uses expert knowledge in the form of manual annotations. Recently, there has been interest in introducing other forms of expert knowledge into deep learning architecture design. This is the approach considered in the paper where we propose to combine ultrasound video with point-of-gaze tracked for expert sonographers as they scan to train memory-efficient ultrasound image analysis models. Specifically we develop teacher-student knowledge transfer models for the exemplar task of frame classification for the fetal abdomen, head, and femur. The best performing memory-efficient models attain performance within 5% of conventional models that are 1000× larger in size
- …