4,789 research outputs found
Learning Input and Recurrent Weight Matrices in Echo State Networks
Abstract The traditional echo state network (ESN) is a special type of a temporally deep model, the recurrent network (RNN), which carefully designs the recurrent matrix and fixes both the recurrent and input matrices in the RNN. The ESN also adopts the linear output (or readout) units to simplify the leanring of the only output matrix in the RNN. In this paper, we devise a special technique that takes advantage of the linearity in the output units in the ESN to learn the input and recurrent matrices, not carried on earlier ESNs due to the well-known difficulty of their learning. Compared with the technique of BackProp Through Time (BPTT) in learning the general RNNs, our proposed technique makes use of the linearity in the output units to provide constraints among various matrices in the RNN, enabling the computation of the gradients as the learning signal in an analytical form instead of by recursion as in the BPTT. Experimental results on phone state classification show that learning either or both the input and recurrent matrices in the ESN is superior to the traditional ESN without learning them, especially when longer time steps are used in analytically computing the gradients
Hierarchical Temporal Representation in Linear Reservoir Computing
Recently, studies on deep Reservoir Computing (RC) highlighted the role of
layering in deep recurrent neural networks (RNNs). In this paper, the use of
linear recurrent units allows us to bring more evidence on the intrinsic
hierarchical temporal representation in deep RNNs through frequency analysis
applied to the state signals. The potentiality of our approach is assessed on
the class of Multiple Superimposed Oscillator tasks. Furthermore, our
investigation provides useful insights to open a discussion on the main aspects
that characterize the deep learning framework in the temporal domain.Comment: This is a pre-print of the paper submitted to the 27th Italian
Workshop on Neural Networks, WIRN 201
Echo State Networks: analysis, training and predictive control
The goal of this paper is to investigate the theoretical properties, the
training algorithm, and the predictive control applications of Echo State
Networks (ESNs), a particular kind of Recurrent Neural Networks. First, a
condition guaranteeing incremetal global asymptotic stability is devised. Then,
a modified training algorithm allowing for dimensionality reduction of ESNs is
presented. Eventually, a model predictive controller is designed to solve the
tracking problem, relying on ESNs as the model of the system. Numerical results
concerning the predictive control of a nonlinear process for pH neutralization
confirm the effectiveness of the proposed algorithms for the identification,
dimensionality reduction, and the control design for ESNs.Comment: 6 pages,5 figures, submitted to European Control Conference (ECC
Echo State Condition at the Critical Point
Recurrent networks with transfer functions that fulfill the Lipschitz
continuity with K=1 may be echo state networks if certain limitations on the
recurrent connectivity are applied. It has been shown that it is sufficient if
the largest singular value of the recurrent connectivity is smaller than 1. The
main achievement of this paper is a proof under which conditions the network is
an echo state network even if the largest singular value is one. It turns out
that in this critical case the exact shape of the transfer function plays a
decisive role in determining whether the network still fulfills the echo state
condition. In addition, several examples with one neuron networks are outlined
to illustrate effects of critical connectivity. Moreover, within the manuscript
a mathematical definition for a critical echo state network is suggested
Empirical Analysis of the Necessary and Sufficient Conditions of the Echo State Property
The Echo State Network (ESN) is a specific recurrent network, which has
gained popularity during the last years. The model has a recurrent network
named reservoir, that is fixed during the learning process. The reservoir is
used for transforming the input space in a larger space. A fundamental property
that provokes an impact on the model accuracy is the Echo State Property (ESP).
There are two main theoretical results related to the ESP. First, a sufficient
condition for the ESP existence that involves the singular values of the
reservoir matrix. Second, a necessary condition for the ESP. The ESP can be
violated according to the spectral radius value of the reservoir matrix. There
is a theoretical gap between these necessary and sufficient conditions. This
article presents an empirical analysis of the accuracy and the projections of
reservoirs that satisfy this theoretical gap. It gives some insights about the
generation of the reservoir matrix. From previous works, it is already known
that the optimal accuracy is obtained near to the border of stability control
of the dynamics. Then, according to our empirical results, we can see that this
border seems to be closer to the sufficient conditions than to the necessary
conditions of the ESP.Comment: 23 pages, 14 figures, accepted paper for the IEEE IJCNN, 201
The Power of Linear Recurrent Neural Networks
Recurrent neural networks are a powerful means to cope with time series. We
show how a type of linearly activated recurrent neural networks, which we call
predictive neural networks, can approximate any time-dependent function f(t)
given by a number of function values. The approximation can effectively be
learned by simply solving a linear equation system; no backpropagation or
similar methods are needed. Furthermore, the network size can be reduced by
taking only most relevant components. Thus, in contrast to others, our approach
not only learns network weights but also the network architecture. The networks
have interesting properties: They end up in ellipse trajectories in the long
run and allow the prediction of further values and compact representations of
functions. We demonstrate this by several experiments, among them multiple
superimposed oscillators (MSO), robotic soccer, and predicting stock prices.
Predictive neural networks outperform the previous state-of-the-art for the MSO
task with a minimal number of units.Comment: 22 pages, 14 figures and tables, revised implementatio
- …