30,424 research outputs found
Regularisation of Neural Networks by Enforcing Lipschitz Continuity
We investigate the effect of explicitly enforcing the Lipschitz continuity of
neural networks with respect to their inputs. To this end, we provide a simple
technique for computing an upper bound to the Lipschitz constant---for multiple
-norms---of a feed forward neural network composed of commonly used layer
types. Our technique is then used to formulate training a neural network with a
bounded Lipschitz constant as a constrained optimisation problem that can be
solved using projected stochastic gradient methods. Our evaluation study shows
that the performance of the resulting models exceeds that of models trained
with other common regularisers. We also provide evidence that the
hyperparameters are intuitive to tune, demonstrate how the choice of norm for
computing the Lipschitz constant impacts the resulting model, and show that the
performance gains provided by our method are particularly noticeable when only
a small amount of training data is available
Multilayered feed forward Artificial Neural Network model to predict the average summer-monsoon rainfall in India
In the present research, possibility of predicting average summer-monsoon
rainfall over India has been analyzed through Artificial Neural Network models.
In formulating the Artificial Neural Network based predictive model, three
layered networks have been constructed with sigmoid non-linearity. The models
under study are different in the number of hidden neurons. After a thorough
training and test procedure, neural net with three nodes in the hidden layer is
found to be the best predictive model.Comment: 19 pages, 1 table, 3 figure
SCANN: Synthesis of Compact and Accurate Neural Networks
Deep neural networks (DNNs) have become the driving force behind recent
artificial intelligence (AI) research. An important problem with implementing a
neural network is the design of its architecture. Typically, such an
architecture is obtained manually by exploring its hyperparameter space and
kept fixed during training. This approach is time-consuming and inefficient.
Another issue is that modern neural networks often contain millions of
parameters, whereas many applications and devices require small inference
models. However, efforts to migrate DNNs to such devices typically entail a
significant loss of classification accuracy. To address these challenges, we
propose a two-step neural network synthesis methodology, called DR+SCANN, that
combines two complementary approaches to design compact and accurate DNNs. At
the core of our framework is the SCANN methodology that uses three basic
architecture-changing operations, namely connection growth, neuron growth, and
connection pruning, to synthesize feed-forward architectures with arbitrary
structure. SCANN encapsulates three synthesis methodologies that apply a
repeated grow-and-prune paradigm to three architectural starting points.
DR+SCANN combines the SCANN methodology with dataset dimensionality reduction
to alleviate the curse of dimensionality. We demonstrate the efficacy of SCANN
and DR+SCANN on various image and non-image datasets. We evaluate SCANN on
MNIST and ImageNet benchmarks. In addition, we also evaluate the efficacy of
using dimensionality reduction alongside SCANN (DR+SCANN) on nine small to
medium-size datasets. We also show that our synthesis methodology yields neural
networks that are much better at navigating the accuracy vs. energy efficiency
space. This would enable neural network-based inference even on
Internet-of-Things sensors.Comment: 13 pages, 8 figure
- …