4 research outputs found
Hierarchical Composition of Memristive Networks for Real-Time Computing
Advances in materials science have led to physical instantiations of
self-assembled networks of memristive devices and demonstrations of their
computational capability through reservoir computing. Reservoir computing is an
approach that takes advantage of collective system dynamics for real-time
computing. A dynamical system, called a reservoir, is excited with a
time-varying signal and observations of its states are used to reconstruct a
desired output signal. However, such a monolithic assembly limits the
computational power due to signal interdependency and the resulting correlated
readouts. Here, we introduce an approach that hierarchically composes a set of
interconnected memristive networks into a larger reservoir. We use signal
amplification and restoration to reduce reservoir state correlation, which
improves the feature extraction from the input signals. Using the same number
of output signals, such a hierarchical composition of heterogeneous small
networks outperforms monolithic memristive networks by at least 20% on waveform
generation tasks. On the NARMA-10 task, we reduce the error by up to a factor
of 2 compared to homogeneous reservoirs with sigmoidal neurons, whereas single
memristive networks are unable to produce the correct result. Hierarchical
composition is key for solving more complex tasks with such novel nano-scale
hardware
Exploring Transfer Function Nonlinearity in Echo State Networks
Supralinear and sublinear pre-synaptic and dendritic integration is
considered to be responsible for nonlinear computation power of biological
neurons, emphasizing the role of nonlinear integration as opposed to nonlinear
output thresholding. How, why, and to what degree the transfer function
nonlinearity helps biologically inspired neural network models is not fully
understood. Here, we study these questions in the context of echo state
networks (ESN). ESN is a simple neural network architecture in which a fixed
recurrent network is driven with an input signal, and the output is generated
by a readout layer from the measurements of the network states. ESN
architecture enjoys efficient training and good performance on certain
signal-processing tasks, such as system identification and time series
prediction. ESN performance has been analyzed with respect to the connectivity
pattern in the network structure and the input bias. However, the effects of
the transfer function in the network have not been studied systematically.
Here, we use an approach tanh on the Taylor expansion of a frequently used
transfer function, the hyperbolic tangent function, to systematically study the
effect of increasing nonlinearity of the transfer function on the memory,
nonlinear capacity, and signal processing performance of ESN. Interestingly, we
find that a quadratic approximation is enough to capture the computational
power of ESN with tanh function. The results of this study apply to both
software and hardware implementation of ESN.Comment: arXiv admin note: text overlap with arXiv:1502.0071
Theory and Practice of Computing with Excitable Dynamics
Reservoir computing (RC) is a promising paradigm for time series processing. In this paradigm, the desired output is computed by combining measurements of an excitable system that responds to time-dependent exogenous stimuli. The excitable system is called a reservoir and measurements of its state are combined using a readout layer to produce a target output. The power of RC is attributed to an emergent short-term memory in dynamical systems and has been analyzed mathematically for both linear and nonlinear dynamical systems. The theory of RC treats only the macroscopic properties of the reservoir, without reference to the underlying medium it is made of. As a result, RC is particularly attractive for building computational devices using emerging technologies whose structure is not exactly controllable, such as self-assembled nanoscale circuits. RC has lacked a formal framework for performance analysis and prediction that goes beyond memory properties. To provide such a framework, here a mathematical theory of memory and information processing in ordered and disordered linear dynamical systems is developed. This theory analyzes the optimal readout layer for a given task. The focus of the theory is a standard model of RC, the echo state network (ESN). An ESN consists of a fixed recurrent neural network that is driven by an external signal. The dynamics of the network is then combined linearly with readout weights to produce the desired output. The readout weights are calculated using linear regression.
Using an analysis of regression equations, the readout weights can be calculated using only the statistical properties of the reservoir dynamics, the input signal, and the desired output. The readout layer weights can be calculated from a priori knowledge of the desired function to be computed and the weight matrix of the reservoir. This formulation explicitly depends on the input weights, the reservoir weights, and the statistics of the target function. This formulation is used to bound the expected error of the system for a given target function. The effects of input-output correlation and complex network structure in the reservoir on the computational performance of the system have been mathematically characterized. Far from the chaotic regime, ordered linear networks exhibit a homogeneous decay of memory in different dimensions, which keeps the input history coherent. As disorder is introduced in the structure of the network, memory decay becomes inhomogeneous along different dimensions causing decoherence in the input history, and degradation in task-solving performance. Close to the chaotic regime, the ordered systems show loss of temporal information in the input history, and therefore inability to solve tasks. However, by introducing disorder and therefore heterogeneous decay of memory the temporal information of input history is preserved and the task-solving performance is recovered. Thus for systems at the edge of chaos, disordered structure may enhance temporal information processing. Although the current framework only applies to linear systems, in principle it can be used to describe the properties of physical reservoir computing, e.g., photonic RC using short coherence-length light