2,445 research outputs found
SCANN: Synthesis of Compact and Accurate Neural Networks
Deep neural networks (DNNs) have become the driving force behind recent
artificial intelligence (AI) research. An important problem with implementing a
neural network is the design of its architecture. Typically, such an
architecture is obtained manually by exploring its hyperparameter space and
kept fixed during training. This approach is time-consuming and inefficient.
Another issue is that modern neural networks often contain millions of
parameters, whereas many applications and devices require small inference
models. However, efforts to migrate DNNs to such devices typically entail a
significant loss of classification accuracy. To address these challenges, we
propose a two-step neural network synthesis methodology, called DR+SCANN, that
combines two complementary approaches to design compact and accurate DNNs. At
the core of our framework is the SCANN methodology that uses three basic
architecture-changing operations, namely connection growth, neuron growth, and
connection pruning, to synthesize feed-forward architectures with arbitrary
structure. SCANN encapsulates three synthesis methodologies that apply a
repeated grow-and-prune paradigm to three architectural starting points.
DR+SCANN combines the SCANN methodology with dataset dimensionality reduction
to alleviate the curse of dimensionality. We demonstrate the efficacy of SCANN
and DR+SCANN on various image and non-image datasets. We evaluate SCANN on
MNIST and ImageNet benchmarks. In addition, we also evaluate the efficacy of
using dimensionality reduction alongside SCANN (DR+SCANN) on nine small to
medium-size datasets. We also show that our synthesis methodology yields neural
networks that are much better at navigating the accuracy vs. energy efficiency
space. This would enable neural network-based inference even on
Internet-of-Things sensors.Comment: 13 pages, 8 figure
Data-Driven Sparse Structure Selection for Deep Neural Networks
Deep convolutional neural networks have liberated its extraordinary power on
various tasks. However, it is still very challenging to deploy state-of-the-art
models into real-world applications due to their high computational complexity.
How can we design a compact and effective network without massive experiments
and expert knowledge? In this paper, we propose a simple and effective
framework to learn and prune deep models in an end-to-end manner. In our
framework, a new type of parameter -- scaling factor is first introduced to
scale the outputs of specific structures, such as neurons, groups or residual
blocks. Then we add sparsity regularizations on these factors, and solve this
optimization problem by a modified stochastic Accelerated Proximal Gradient
(APG) method. By forcing some of the factors to zero, we can safely remove the
corresponding structures, thus prune the unimportant parts of a CNN. Comparing
with other structure selection methods that may need thousands of trials or
iterative fine-tuning, our method is trained fully end-to-end in one training
pass without bells and whistles. We evaluate our method, Sparse Structure
Selection with several state-of-the-art CNNs, and demonstrate very promising
results with adaptive depth and width selection.Comment: ECCV Camera ready versio
- …