38 research outputs found
The spectral radius remains a valid indicator of the echo state property for large reservoirs
In the field of Reservoir Computing, scaling the spectral radius of the weight matrix of a random recurrent neural network to below unity is a commonly used method to ensure the Echo State Property. Recently it has been shown that this condition is too weak. To overcome this problem, other more involved - sufficient conditions for the Echo State Property have been proposed. In this paper we provide a large-scale experimental verification of the Echo State Property for large recurrent neural networks with zero input and zero bias. Our main conclusion is that the spectral radius method remains a valid indicator of the Echo State Property; the probability that the Echo State Property does not hold, drops for larger networks with spectral radius below unity, which are the ones of practical interest
Product Reservoir Computing: Time-Series Computation with Multiplicative Neurons
Echo state networks (ESN), a type of reservoir computing (RC) architecture,
are efficient and accurate artificial neural systems for time series processing
and learning. An ESN consists of a core of recurrent neural networks, called a
reservoir, with a small number of tunable parameters to generate a
high-dimensional representation of an input, and a readout layer which is
easily trained using regression to produce a desired output from the reservoir
states. Certain computational tasks involve real-time calculation of high-order
time correlations, which requires nonlinear transformation either in the
reservoir or the readout layer. Traditional ESN employs a reservoir with
sigmoid or tanh function neurons. In contrast, some types of biological neurons
obey response curves that can be described as a product unit rather than a sum
and threshold. Inspired by this class of neurons, we introduce a RC
architecture with a reservoir of product nodes for time series computation. We
find that the product RC shows many properties of standard ESN such as
short-term memory and nonlinear capacity. On standard benchmarks for chaotic
prediction tasks, the product RC maintains the performance of a standard
nonlinear ESN while being more amenable to mathematical analysis. Our study
provides evidence that such networks are powerful in highly nonlinear tasks
owing to high-order statistics generated by the recurrent product node
reservoir
Optimal Input Representation in Neural Systems at the Edge of Chaos
Shedding light on how biological systems represent, process and store information in noisy
environments is a key and challenging goal. A stimulating, though controversial, hypothesis poses
that operating in dynamical regimes near the edge of a phase transition, i.e., at criticality or the âedge
of chaosâ, can provide information-processing living systems with important operational advantages,
creating, e.g., an optimal trade-off between robustness and flexibility. Here, we elaborate on a recent
theoretical result, which establishes that the spectrum of covariance matrices of neural networks
representing complex inputs in a robust way needs to decay as a power-law of the rank, with an
exponent close to unity, a result that has been indeed experimentally verified in neurons of the mouse
visual cortex. Aimed at understanding and mimicking these results, we construct an artificial neural
network and train it to classify images. We find that the best performance in such a task is obtained
when the network operates near the critical point, at which the eigenspectrum of the covariance
matrix follows the very same statistics as actual neurons do. Thus, we conclude that operating near
criticality can also haveâbesides the usually alleged virtuesâthe advantage of allowing for flexible,
robust and efficient input representations.The Spanish Ministry and Agencia Estatal de investigaciĂłn
(AEI) through grant FIS2017-84256-P (European Regional Development Fund)âConsejerĂa de Conocimiento, InvestigaciĂłn Universidad, Junta de AndalucĂaâ and European Regional
Development Fund, Project Ref. A-FQM-175-UGR18 and Project Ref. P20-0017
Predicting Shallow Water Dynamics using Echo-State Networks with Transfer Learning
In this paper we demonstrate that reservoir computing can be used to learn
the dynamics of the shallow-water equations. In particular, while most previous
applications of reservoir computing have required training on a particular
trajectory to further predict the evolution along that trajectory alone, we
show the capability of reservoir computing to predict trajectories of the
shallow-water equations with initial conditions not seen in the training
process. However, in this setting, we find that the performance of the network
deteriorates for initial conditions with ambient conditions (such as total
water height and average velocity) that are different from those in the
training dataset. To circumvent this deficiency, we introduce a transfer
learning approach wherein a small additional training step with the relevant
ambient conditions is used to improve the predictions
Echo State Property of Deep Reservoir Computing Networks
In the last years, the Reservoir Computing (RC) framework has emerged as a state of-the-art approach for efficient learning in temporal domains. Recently, within the RC context, deep Echo State Network (ESN) models have been proposed. Being composed of a stack of multiple non-linear reservoir layers, deep ESNs potentially allow to exploit the advantages of a hierarchical temporal feature representation at different levels of abstraction, at the same time preserving the training efficiency typical of the RC methodology. In this paper, we generalize to the case of deep architectures the fundamental RC conditions related to the Echo State Property (ESP), based on the study of stability and contractivity of the resulting dynamical system. Besides providing a necessary condition and a sufficient condition for the ESP of layered RC networks, the results of our analysis provide also insights on the nature of the state dynamics in hierarchically organized recurrent models. In particular, we find out that by adding layers to a deep reservoir architecture, the regime of networkâs dynamics can only be driven towards (equally or) less stable behaviors. Moreover, our investigation shows the intrinsic ability of temporal dynamics differentiation at the different levels in a deep recurrent architecture, with higher layers in the stack characterized by less contractive dynamics. Such theoretical insights are further supported by experimental results that show the effect of layering in terms of a progressively increased short-term memory capacity of the recurrent models