4 research outputs found
Feature Fusion for Online Mutual Knowledge Distillation
We propose a learning framework named Feature Fusion Learning (FFL) that
efficiently trains a powerful classifier through a fusion module which combines
the feature maps generated from parallel neural networks. Specifically, we
train a number of parallel neural networks as sub-networks, then we combine the
feature maps from each sub-network using a fusion module to create a more
meaningful feature map. The fused feature map is passed into the fused
classifier for overall classification. Unlike existing feature fusion methods,
in our framework, an ensemble of sub-network classifiers transfers its
knowledge to the fused classifier and then the fused classifier delivers its
knowledge back to each sub-network, mutually teaching one another in an
online-knowledge distillation manner. This mutually teaching system not only
improves the performance of the fused classifier but also obtains performance
gain in each sub-network. Moreover, our model is more beneficial because
different types of network can be used for each sub-network. We have performed
a variety of experiments on multiple datasets such as CIFAR-10, CIFAR-100 and
ImageNet and proved that our method is more effective than other alternative
methods in terms of performance of both sub-networks and the fused classifier.Comment: International Conference on Pattern Recognitio
Peer Collaborative Learning for Polyphonic Sound Event Detection
This paper describes that semi-supervised learning called peer collaborative
learning (PCL) can be applied to the polyphonic sound event detection (PSED)
task, which is one of the tasks in the Detection and Classification of Acoustic
Scenes and Events (DCASE) challenge. Many deep learning models have been
studied to find out what kind of sound events occur where and for how long in a
given audio clip. The characteristic of PCL used in this paper is the
combination of ensemble-based knowledge distillation into sub-networks and
student-teacher model-based knowledge distillation, which can train a robust
PSED model from a small amount of strongly labeled data, weakly labeled data,
and a large amount of unlabeled data. We evaluated the proposed PCL model using
the DCASE 2019 Task 4 datasets and achieved an F1-score improvement of about
10% compared to the baseline model.Comment: Submitted to ICASSP 202
Decision Support for Video-based Detection of Flu Symptoms
The development of decision support systems is a growing domain that can be
applied in the area of disease control and diagnostics. Using video-based
surveillance data, skeleton features are extracted to perform action
recognition, specifically the detection and recognition of coughing and
sneezing motions. Providing evidence of flu-like symptoms, a decision support
system based on causal networks is capable of providing the operator with vital
information for decision-making. A modified residual temporal convolutional
network is proposed for action recognition using skeleton features. This paper
addresses the capability of using results from a machine-learning model as
evidence for a cognitive decision support system. We propose risk and trust
measures as a metric to bridge between machine-learning and machine-reasoning.
We provide experiments on evaluating the performance of the proposed network
and how these performance measures can be combined with risk to generate trust.Comment: 8 pages, 7 figures, submitted to IEEE SM
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
Deep neural models in recent years have been successful in almost every
field, including extremely complex problem statements. However, these models
are huge in size, with millions (and even billions) of parameters, thus
demanding more heavy computation power and failing to be deployed on edge
devices. Besides, the performance boost is highly dependent on redundant
labeled data. To achieve faster speeds and to handle the problems caused by the
lack of data, knowledge distillation (KD) has been proposed to transfer
information learned from one model to another. KD is often characterized by the
so-called `Student-Teacher' (S-T) learning framework and has been broadly
applied in model compression and knowledge transfer. This paper is about KD and
S-T learning, which are being actively studied in recent years. First, we aim
to provide explanations of what KD is and how/why it works. Then, we provide a
comprehensive survey on the recent progress of KD methods together with S-T
frameworks typically for vision tasks. In general, we consider some fundamental
questions that have been driving this research area and thoroughly generalize
the research progress and technical details. Additionally, we systematically
analyze the research status of KD in vision applications. Finally, we discuss
the potentials and open challenges of existing methods and prospect the future
directions of KD and S-T learning.Comment: Accepted to IEEE Transactions on Pattern Analysis and Machine
Intelligence(TPAMI),202