33,015 research outputs found
Incremental construction of LSTM recurrent neural network
Long Short--Term Memory (LSTM) is a recurrent neural network that
uses structures called memory blocks to allow the net remember
significant events distant in the past input sequence in order to
solve long time lag tasks, where other RNN approaches fail.
Throughout this work we have performed experiments using LSTM
networks extended with growing abilities, which we call GLSTM.
Four methods of training growing LSTM has been compared. These
methods include cascade and fully connected hidden layers as well
as two different levels of freezing previous weights in the
cascade case. GLSTM has been applied to a forecasting problem in a biomedical domain, where the input/output behavior of five
controllers of the Central Nervous System control has to be
modelled. We have compared growing LSTM results against other
neural networks approaches, and our work applying conventional
LSTM to the task at hand.Postprint (published version
Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing
This work presents a new algorithm called evolutionary exploration of
augmenting convolutional topologies (EXACT), which is capable of evolving the
structure of convolutional neural networks (CNNs). EXACT is in part modeled
after the neuroevolution of augmenting topologies (NEAT) algorithm, with
notable exceptions to allow it to scale to large scale distributed computing
environments and evolve networks with convolutional filters. In addition to
multithreaded and MPI versions, EXACT has been implemented as part of a BOINC
volunteer computing project, allowing large scale evolution. During a period of
two months, over 4,500 volunteered computers on the Citizen Science Grid
trained over 120,000 CNNs and evolved networks reaching 98.32% test data
accuracy on the MNIST handwritten digits dataset. These results are even
stronger as the backpropagation strategy used to train the CNNs was fairly
rudimentary (ReLU units, L2 regularization and Nesterov momentum) and these
were initial test runs done without refinement of the backpropagation
hyperparameters. Further, the EXACT evolutionary strategy is independent of the
method used to train the CNNs, so they could be further improved by advanced
techniques like elastic distortions, pretraining and dropout. The evolved
networks are also quite interesting, showing "organic" structures and
significant differences from standard human designed architectures.Comment: 17 pages, 13 figures. Submitted to the 2017 Genetic and Evolutionary
Computation Conference (GECCO 2017
An Adaptive Locally Connected Neuron Model: Focusing Neuron
This paper presents a new artificial neuron model capable of learning its
receptive field in the topological domain of inputs. The model provides
adaptive and differentiable local connectivity (plasticity) applicable to any
domain. It requires no other tool than the backpropagation algorithm to learn
its parameters which control the receptive field locations and apertures. This
research explores whether this ability makes the neuron focus on informative
inputs and yields any advantage over fully connected neurons. The experiments
include tests of focusing neuron networks of one or two hidden layers on
synthetic and well-known image recognition data sets. The results demonstrated
that the focusing neurons can move their receptive fields towards more
informative inputs. In the simple two-hidden layer networks, the focusing
layers outperformed the dense layers in the classification of the 2D spatial
data sets. Moreover, the focusing networks performed better than the dense
networks even when 70 of the weights were pruned. The tests on
convolutional networks revealed that using focusing layers instead of dense
layers for the classification of convolutional features may work better in some
data sets.Comment: 45 pages, a national patent filed, submitted to Turkish Patent
Office, No: -2017/17601, Date: 09.11.201
- …