4,256 research outputs found
Incremental construction of LSTM recurrent neural network
Long Short--Term Memory (LSTM) is a recurrent neural network that
uses structures called memory blocks to allow the net remember
significant events distant in the past input sequence in order to
solve long time lag tasks, where other RNN approaches fail.
Throughout this work we have performed experiments using LSTM
networks extended with growing abilities, which we call GLSTM.
Four methods of training growing LSTM has been compared. These
methods include cascade and fully connected hidden layers as well
as two different levels of freezing previous weights in the
cascade case. GLSTM has been applied to a forecasting problem in a biomedical domain, where the input/output behavior of five
controllers of the Central Nervous System control has to be
modelled. We have compared growing LSTM results against other
neural networks approaches, and our work applying conventional
LSTM to the task at hand.Postprint (published version
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net),
a very deep architecture where each stacked processing block is derived from a
time-invariant non-autonomous dynamical system. Non-autonomy is implemented by
skip connections from the block input to each of the unrolled processing stages
and allows stability to be enforced so that blocks can be unrolled adaptively
to a pattern-dependent processing depth. NAIS-Net induces non-trivial,
Lipschitz input-output maps, even for an infinite unroll length. We prove that
the network is globally asymptotically stable so that for every initial
condition there is exactly one input-dependent equilibrium assuming tanh units,
and multiple stable equilibria for ReL units. An efficient implementation that
enforces the stability under derived conditions for both fully-connected and
convolutional layers is also presented. Experimental results show how NAIS-Net
exhibits stability in practice, yielding a significant reduction in
generalization gap compared to ResNets.Comment: NIPS 201
On the Stability of Gated Graph Neural Networks
In this paper, we aim to find the conditions for input-state stability (ISS)
and incremental input-state stability (ISS) of Gated Graph Neural
Networks (GGNNs). We show that this recurrent version of Graph Neural Networks
(GNNs) can be expressed as a dynamical distributed system and, as a
consequence, can be analysed using model-based techniques to assess its
stability and robustness properties. Then, the stability criteria found can be
exploited as constraints during the training process to enforce the internal
stability of the neural network. Two distributed control examples, flocking and
multi-robot motion control, show that using these conditions increases the
performance and robustness of the gated GNNs
- …