17 research outputs found
Balanced Sparsity for Efficient DNN Inference on GPU
In trained deep neural networks, unstructured pruning can reduce redundant
weights to lower storage cost. However, it requires the customization of
hardwares to speed up practical inference. Another trend accelerates sparse
model inference on general-purpose hardwares by adopting coarse-grained
sparsity to prune or regularize consecutive weights for efficient computation.
But this method often sacrifices model accuracy. In this paper, we propose a
novel fine-grained sparsity approach, balanced sparsity, to achieve high model
accuracy with commercial hardwares efficiently. Our approach adapts to high
parallelism property of GPU, showing incredible potential for sparsity in the
widely deployment of deep learning services. Experiment results show that
balanced sparsity achieves up to 3.1x practical speedup for model inference on
GPU, while retains the same high model accuracy as fine-grained sparsity
A genetic algorithm to obtain the optimal recurrent neural network
AbstractSelecting the optimal topology of a neural network for a particular application is a difficult task. In the case of recurrent neural networks, most methods only induce topologies in which their neurons are fully connected. In this paper, we present a genetic algorithm capable of obtaining not only the optimal topology of a recurrent neural network but also the least number of connections necessary. Finally, this genetic algorithm is applied to a problem of grammatical inference using neural networks, with very good results
Stability and Memory-loss go Hand-in-Hand: Three Results in Dynamics & Computation
The search for universal laws that help establish a relationship between
dynamics and computation is driven by recent expansionist initiatives in
biologically inspired computing. A general setting to understand both such
dynamics and computation is a driven dynamical system that responds to a
temporal input. Surprisingly, we find memory-loss a feature of driven systems
to forget their internal states helps provide unambiguous answers to the
following fundamental stability questions that have been unanswered for
decades: what is necessary and sufficient so that slightly different inputs
still lead to mostly similar responses? How does changing the driven system's
parameters affect stability? What is the mathematical definition of the
edge-of-criticality? We anticipate our results to be timely in understanding
and designing biologically inspired computers that are entering an era of
dedicated hardware implementations for neuromorphic computing and
state-of-the-art reservoir computing applications.Comment: To appear in the Proceedings of the Royal Society of London, Series
Meta-Learning Evolutionary Artificial Neural Networks
In this paper, we present MLEANN (Meta-Learning Evolutionary Artificial
Neural Network), an automatic computational framework for the adaptive
optimization of artificial neural networks wherein the neural network
architecture, activation function, connection weights; learning algorithm and
its parameters are adapted according to the problem. We explored the
performance of MLEANN and conventionally designed artificial neural networks
for function approximation problems. To evaluate the comparative performance,
we used three different well-known chaotic time series. We also present the
state of the art popular neural network learning algorithms and some
experimentation results related to convergence speed and generalization
performance. We explored the performance of backpropagation algorithm;
conjugate gradient algorithm, quasi-Newton algorithm and Levenberg-Marquardt
algorithm for the three chaotic time series. Performances of the different
learning algorithms were evaluated when the activation functions and
architecture were changed. We further present the theoretical background,
algorithm, design strategy and further demonstrate how effective and inevitable
is the proposed MLEANN framework to design a neural network, which is smaller,
faster and with a better generalization performance