13 research outputs found
The Asymptotic Performance of Linear Echo State Neural Networks
In this article, a study of the mean-square error (MSE) performance of linear
echo-state neural networks is performed, both for training and testing tasks.
Considering the realistic setting of noise present at the network nodes, we
derive deterministic equivalents for the aforementioned MSE in the limit where
the number of input data and network size both grow large. Specializing
then the network connectivity matrix to specific random settings, we further
obtain simple formulas that provide new insights on the performance of such
networks
A Random Matrix Approach to Echo-State Neural Networks
Abstract Recurrent neural networks, especially in their linear version, have provided many qualitative insights on their performance under different configurations. This article provides, through a novel random matrix framework, the quantitative counterpart of these performance results, specifically in the case of echo-state networks. Beyond mere insights, our approach conveys a deeper understanding on the core mechanism under play for both training and testing
Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systems
A new class of non-homogeneous state-affine systems is introduced for use in
reservoir computing. Sufficient conditions are identified that guarantee first,
that the associated reservoir computers with linear readouts are causal,
time-invariant, and satisfy the fading memory property and second, that a
subset of this class is universal in the category of fading memory filters with
stochastic almost surely uniformly bounded inputs. This means that any
discrete-time filter that satisfies the fading memory property with random
inputs of that type can be uniformly approximated by elements in the
non-homogeneous state-affine family.Comment: 41 page
Dynamical systems as temporal feature spaces
Parameterized state space models in the form of recurrent networks are often
used in machine learning to learn from data streams exhibiting temporal
dependencies. To break the black box nature of such models it is important to
understand the dynamical features of the input driving time series that are
formed in the state space. We propose a framework for rigorous analysis of such
state representations in vanishing memory state space models such as echo state
networks (ESN). In particular, we consider the state space a temporal feature
space and the readout mapping from the state space a kernel machine operating
in that feature space. We show that: (1) The usual ESN strategy of randomly
generating input-to-state, as well as state coupling leads to shallow memory
time series representations, corresponding to cross-correlation operator with
fast exponentially decaying coefficients; (2) Imposing symmetry on dynamic
coupling yields a constrained dynamic kernel matching the input time series
with straightforward exponentially decaying motifs or exponentially decaying
motifs of the highest frequency; (3) Simple cycle high-dimensional reservoir
topology specified only through two free parameters can implement deep memory
dynamic kernels with a rich variety of matching motifs. We quantify richness of
feature representations imposed by dynamic kernels and demonstrate that for
dynamic kernel associated with cycle reservoir topology, the kernel richness
undergoes a phase transition close to the edge of stability.Comment: 45 pages, 17 figures, accepte