48 research outputs found
Multi-task zipping via layer-wise neuron sharing
Future mobile devices are anticipated to perceive, understand and react to
the world on their own by running multiple correlated deep neural networks
on-device. Yet the complexity of these neural networks needs to be trimmed down
both within-model and cross-model to fit in mobile storage and memory. Previous
studies focus on squeezing the redundancy within a single neural network. In
this work, we aim to reduce the redundancy across multiple models. We propose
Multi-Task Zipping (MTZ), a framework to automatically merge correlated,
pre-trained deep neural networks for cross-model compression. Central in MTZ is
a layer-wise neuron sharing and incoming weight updating scheme that induces a
minimal change in the error function. MTZ inherits information from each model
and demands light retraining to re-boost the accuracy of individual tasks.
Evaluations show that MTZ is able to fully merge the hidden layers of two
VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet
and CelebA, or share 39.61% parameters between the two networks with <0.5%
increase in the test errors for both tasks. The number of iterations to retrain
the combined network is at least 17.8 times lower than that of training a
single VGG-16 network. Moreover, experiments show that MTZ is also able to
effectively merge multiple residual networks.Comment: Published as a conference paper at NeurIPS 201
MIMONet: Multi-Input Multi-Output On-Device Deep Learning
Future intelligent robots are expected to process multiple inputs
simultaneously (such as image and audio data) and generate multiple outputs
accordingly (such as gender and emotion), similar to humans. Recent research
has shown that multi-input single-output (MISO) deep neural networks (DNN)
outperform traditional single-input single-output (SISO) models, representing a
significant step towards this goal. In this paper, we propose MIMONet, a novel
on-device multi-input multi-output (MIMO) DNN framework that achieves high
accuracy and on-device efficiency in terms of critical performance metrics such
as latency, energy, and memory usage. Leveraging existing SISO model
compression techniques, MIMONet develops a new deep-compression method that is
specifically tailored to MIMO models. This new method explores unique yet
non-trivial properties of the MIMO model, resulting in boosted accuracy and
on-device efficiency. Extensive experiments on three embedded platforms
commonly used in robotic systems, as well as a case study using the TurtleBot3
robot, demonstrate that MIMONet achieves higher accuracy and superior on-device
efficiency compared to state-of-the-art SISO and MISO models, as well as a
baseline MIMO model we constructed. Our evaluation highlights the real-world
applicability of MIMONet and its potential to significantly enhance the
performance of intelligent robotic systems.Comment: Submitted to ICRA 202
Recommended from our members
Multi-task Pruning via Filter Index Sharing: A Many-Objective Optimization Approach
© The Author(s) 2021. State-of-the-art deep neural network plays an increasingly important role in artificial intelligence, while the huge number of parameters in networks brings high memory cost and computational complexity. To solve this problem, filter pruning is widely used for neural network compression and acceleration. However, existing algorithms focus mainly on pruning single model, and few results are available to multi-task pruning that is capable of pruning multi-model and promoting the learning performance. By utilizing the filter sharing technique, this paper aimed to establish a multi-task pruning framework for simultaneously pruning and merging filters in multi-task networks. An optimization problem of selecting the important filters is solved by developing a many-objective optimization algorithm where three criteria are adopted as objectives for the many-objective optimization problem. With the purpose of keeping the network structure, an index matrix is introduced to regulate the information sharing during multi-task training. The proposed multi-task pruning algorithm is quite flexible that can be performed with either adaptive or pre-specified pruning rates. Extensive experiments are performed to verify the applicability and superiority of the proposed method on both single-task and multi-task pruning.National Natural Science Foundation of China under Grants 61701238, 11431015, 61773209, 61873148 and 61933007; Natural Science Foundation of Jiangsu Province of China under Grant BK20190021; Six Talent Peaks Project in Jiangsu Province of China under Grant XYDXX-033; Royal Society of the U.K.; Alexander von Humboldt Foundation of Germany
REPAIR: REnormalizing Permuted Activations for Interpolation Repair
In this paper we look into the conjecture of Entezari et al. (2021) which
states that if the permutation invariance of neural networks is taken into
account, then there is likely no loss barrier to the linear interpolation
between SGD solutions. First, we observe that neuron alignment methods alone
are insufficient to establish low-barrier linear connectivity between SGD
solutions due to a phenomenon we call variance collapse: interpolated deep
networks suffer a collapse in the variance of their activations, causing poor
performance. Next, we propose REPAIR (REnormalizing Permuted Activations for
Interpolation Repair) which mitigates variance collapse by rescaling the
preactivations of such interpolated networks. We explore the interaction
between our method and the choice of normalization layer, network width, and
depth, and demonstrate that using REPAIR on top of neuron alignment methods
leads to 60%-100% relative barrier reduction across a wide variety of
architecture families and tasks. In particular, we report a 74% barrier
reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on
CIFAR10