80,803 research outputs found
Reservoir Topology in Deep Echo State Networks
Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix
Reservoir Topology in Deep Echo State Networks
Deep Echo State Networks (DeepESNs) recently extended the applicability of
Reservoir Computing (RC) methods towards the field of deep learning. In this
paper we study the impact of constrained reservoir topologies in the
architectural design of deep reservoirs, through numerical experiments on
several RC benchmarks. The major outcome of our investigation is to show the
remarkable effect, in terms of predictive performance gain, achieved by the
synergy between a deep reservoir construction and a structured organization of
the recurrent units in each layer. Our results also indicate that a
particularly advantageous architectural setting is obtained in correspondence
of DeepESNs where reservoir units are structured according to a permutation
recurrent matrix.Comment: Preprint of the paper published in the proceedings of ICANN 201
Richness of Deep Echo State Network Dynamics
Reservoir Computing (RC) is a popular methodology for the efficient design of
Recurrent Neural Networks (RNNs). Recently, the advantages of the RC approach
have been extended to the context of multi-layered RNNs, with the introduction
of the Deep Echo State Network (DeepESN) model. In this paper, we study the
quality of state dynamics in progressively higher layers of DeepESNs, using
tools from the areas of information theory and numerical analysis. Our
experimental results on RC benchmark datasets reveal the fundamental role
played by the strength of inter-reservoir connections to increasingly enrich
the representations developed in higher layers. Our analysis also gives
interesting insights into the possibility of effective exploitation of training
algorithms based on stochastic gradient descent in the RC field.Comment: Preprint of the paper accepted at IWANN 201
Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness
This paper introduces recurrent equilibrium networks (RENs), a new class of
nonlinear dynamical models for applications in machine learning, system
identification and control. The new model class has ``built in'' guarantees of
stability and robustness: all models in the class are contracting - a strong
form of nonlinear stability - and models can satisfy prescribed incremental
integral quadratic constraints (IQC), including Lipschitz bounds and
incremental passivity. RENs are otherwise very flexible: they can represent all
stable linear systems, all previously-known sets of contracting recurrent
neural networks and echo state networks, all deep feedforward neural networks,
and all stable Wiener/Hammerstein models. RENs are parameterized directly by a
vector in R^N, i.e. stability and robustness are ensured without parameter
constraints, which simplifies learning since generic methods for unconstrained
optimization can be used. The performance and robustness of the new model set
is evaluated on benchmark nonlinear system identification problems, and the
paper also presents applications in data-driven nonlinear observer design and
control with stability guarantees.Comment: Journal submission, extended version of conference paper (v1 of this
arxiv preprint
Hierarchical Temporal Representation in Linear Reservoir Computing
Recently, studies on deep Reservoir Computing (RC) highlighted the role of
layering in deep recurrent neural networks (RNNs). In this paper, the use of
linear recurrent units allows us to bring more evidence on the intrinsic
hierarchical temporal representation in deep RNNs through frequency analysis
applied to the state signals. The potentiality of our approach is assessed on
the class of Multiple Superimposed Oscillator tasks. Furthermore, our
investigation provides useful insights to open a discussion on the main aspects
that characterize the deep learning framework in the temporal domain.Comment: This is a pre-print of the paper submitted to the 27th Italian
Workshop on Neural Networks, WIRN 201
Echo State Networks: analysis, training and predictive control
The goal of this paper is to investigate the theoretical properties, the
training algorithm, and the predictive control applications of Echo State
Networks (ESNs), a particular kind of Recurrent Neural Networks. First, a
condition guaranteeing incremetal global asymptotic stability is devised. Then,
a modified training algorithm allowing for dimensionality reduction of ESNs is
presented. Eventually, a model predictive controller is designed to solve the
tracking problem, relying on ESNs as the model of the system. Numerical results
concerning the predictive control of a nonlinear process for pH neutralization
confirm the effectiveness of the proposed algorithms for the identification,
dimensionality reduction, and the control design for ESNs.Comment: 6 pages,5 figures, submitted to European Control Conference (ECC
- …