96 research outputs found
Towards a Calculus of Echo State Networks
Reservoir computing is a recent trend in neural networks which uses the
dynamical perturbations on the phase space of a system to compute a desired
target function. We present how one can formulate an expectation of system
performance in a simple class of reservoir computing called echo state
networks. In contrast with previous theoretical frameworks, which only reveal
an upper bound on the total memory in the system, we analytically calculate the
entire memory curve as a function of the structure of the system and the
properties of the input and the target function. We demonstrate the precision
of our framework by validating its result for a wide range of system sizes and
spectral radii. Our analytical calculation agrees with numerical simulations.
To the best of our knowledge this work presents the first exact analytical
characterization of the memory curve in echo state networks
Exploring Transfer Function Nonlinearity in Echo State Networks
Supralinear and sublinear pre-synaptic and dendritic integration is
considered to be responsible for nonlinear computation power of biological
neurons, emphasizing the role of nonlinear integration as opposed to nonlinear
output thresholding. How, why, and to what degree the transfer function
nonlinearity helps biologically inspired neural network models is not fully
understood. Here, we study these questions in the context of echo state
networks (ESN). ESN is a simple neural network architecture in which a fixed
recurrent network is driven with an input signal, and the output is generated
by a readout layer from the measurements of the network states. ESN
architecture enjoys efficient training and good performance on certain
signal-processing tasks, such as system identification and time series
prediction. ESN performance has been analyzed with respect to the connectivity
pattern in the network structure and the input bias. However, the effects of
the transfer function in the network have not been studied systematically.
Here, we use an approach tanh on the Taylor expansion of a frequently used
transfer function, the hyperbolic tangent function, to systematically study the
effect of increasing nonlinearity of the transfer function on the memory,
nonlinear capacity, and signal processing performance of ESN. Interestingly, we
find that a quadratic approximation is enough to capture the computational
power of ESN with tanh function. The results of this study apply to both
software and hardware implementation of ESN.Comment: arXiv admin note: text overlap with arXiv:1502.0071
Product Reservoir Computing: Time-Series Computation with Multiplicative Neurons
Echo state networks (ESN), a type of reservoir computing (RC) architecture,
are efficient and accurate artificial neural systems for time series processing
and learning. An ESN consists of a core of recurrent neural networks, called a
reservoir, with a small number of tunable parameters to generate a
high-dimensional representation of an input, and a readout layer which is
easily trained using regression to produce a desired output from the reservoir
states. Certain computational tasks involve real-time calculation of high-order
time correlations, which requires nonlinear transformation either in the
reservoir or the readout layer. Traditional ESN employs a reservoir with
sigmoid or tanh function neurons. In contrast, some types of biological neurons
obey response curves that can be described as a product unit rather than a sum
and threshold. Inspired by this class of neurons, we introduce a RC
architecture with a reservoir of product nodes for time series computation. We
find that the product RC shows many properties of standard ESN such as
short-term memory and nonlinear capacity. On standard benchmarks for chaotic
prediction tasks, the product RC maintains the performance of a standard
nonlinear ESN while being more amenable to mathematical analysis. Our study
provides evidence that such networks are powerful in highly nonlinear tasks
owing to high-order statistics generated by the recurrent product node
reservoir
Hierarchical Composition of Memristive Networks for Real-Time Computing
Advances in materials science have led to physical instantiations of
self-assembled networks of memristive devices and demonstrations of their
computational capability through reservoir computing. Reservoir computing is an
approach that takes advantage of collective system dynamics for real-time
computing. A dynamical system, called a reservoir, is excited with a
time-varying signal and observations of its states are used to reconstruct a
desired output signal. However, such a monolithic assembly limits the
computational power due to signal interdependency and the resulting correlated
readouts. Here, we introduce an approach that hierarchically composes a set of
interconnected memristive networks into a larger reservoir. We use signal
amplification and restoration to reduce reservoir state correlation, which
improves the feature extraction from the input signals. Using the same number
of output signals, such a hierarchical composition of heterogeneous small
networks outperforms monolithic memristive networks by at least 20% on waveform
generation tasks. On the NARMA-10 task, we reduce the error by up to a factor
of 2 compared to homogeneous reservoirs with sigmoidal neurons, whereas single
memristive networks are unable to produce the correct result. Hierarchical
composition is key for solving more complex tasks with such novel nano-scale
hardware
Computational Capacity and Energy Consumption of Complex Resistive Switch Networks
Resistive switches are a class of emerging nanoelectronics devices that
exhibit a wide variety of switching characteristics closely resembling
behaviors of biological synapses. Assembled into random networks, such
resistive switches produce emerging behaviors far more complex than that of
individual devices. This was previously demonstrated in simulations that
exploit information processing within these random networks to solve tasks that
require nonlinear computation as well as memory. Physical assemblies of such
networks manifest complex spatial structures and basic processing capabilities
often related to biologically-inspired computing. We model and simulate random
resistive switch networks and analyze their computational capacities. We
provide a detailed discussion of the relevant design parameters and establish
the link to the physical assemblies by relating the modeling parameters to
physical parameters. More globally connected networks and an increased network
switching activity are means to increase the computational capacity linearly at
the expense of exponentially growing energy consumption. We discuss a new
modular approach that exhibits higher computational capacities and energy
consumption growing linearly with the number of networks used. The results show
how to optimize the trade-off between computational capacity and energy
efficiency and are relevant for the design and fabrication of novel computing
architectures that harness random assemblies of emerging nanodevices
- …