Exploring Transfer Function Nonlinearity in Echo State Networks

Supralinear and sublinear pre-synaptic and dendritic integration is considered to be responsible for nonlinear computation power of biological neurons, emphasizing the role of nonlinear integration as opposed to nonlinear output thresholding. How, why, and to what degree the transfer function nonlinearity helps biologically inspired neural network models is not fully understood. Here, we study these questions in the context of echo state networks (ESN). ESN is a simple neural network architecture in which a fixed recurrent network is driven with an input signal, and the output is generated by a readout layer from the measurements of the network states. ESN architecture enjoys efficient training and good performance on certain signal-processing tasks, such as system identification and time series prediction. ESN performance has been analyzed with respect to the connectivity pattern in the network structure and the input bias. However, the effects of the transfer function in the network have not been studied systematically. Here, we use an approach tanh on the Taylor expansion of a frequently used transfer function, the hyperbolic tangent function, to systematically study the effect of increasing nonlinearity of the transfer function on the memory, nonlinear capacity, and signal processing performance of ESN. Interestingly, we find that a quadratic approximation is enough to capture the computational power of ESN with tanh function. The results of this study apply to both software and hardware implementation of ESN.Comment: arXiv admin note: text overlap with arXiv:1502.0071

Product Reservoir Computing: Time-Series Computation with Multiplicative Neurons

Echo state networks (ESN), a type of reservoir computing (RC) architecture, are efficient and accurate artificial neural systems for time series processing and learning. An ESN consists of a core of recurrent neural networks, called a reservoir, with a small number of tunable parameters to generate a high-dimensional representation of an input, and a readout layer which is easily trained using regression to produce a desired output from the reservoir states. Certain computational tasks involve real-time calculation of high-order time correlations, which requires nonlinear transformation either in the reservoir or the readout layer. Traditional ESN employs a reservoir with sigmoid or tanh function neurons. In contrast, some types of biological neurons obey response curves that can be described as a product unit rather than a sum and threshold. Inspired by this class of neurons, we introduce a RC architecture with a reservoir of product nodes for time series computation. We find that the product RC shows many properties of standard ESN such as short-term memory and nonlinear capacity. On standard benchmarks for chaotic prediction tasks, the product RC maintains the performance of a standard nonlinear ESN while being more amenable to mathematical analysis. Our study provides evidence that such networks are powerful in highly nonlinear tasks owing to high-order statistics generated by the recurrent product node reservoir

Potential implementation of Reservoir Computing models based on magnetic skyrmions

Reservoir Computing is a type of recursive neural network commonly used for recognizing and predicting spatio-temporal events relying on a complex hierarchy of nested feedback loops to generate a memory functionality. The Reservoir Computing paradigm does not require any knowledge of the reservoir topology or node weights for training purposes and can therefore utilize naturally existing networks formed by a wide variety of physical processes. Most efforts prior to this have focused on utilizing memristor techniques to implement recursive neural networks. This paper examines the potential of skyrmion fabrics formed in magnets with broken inversion symmetry that may provide an attractive physical instantiation for Reservoir Computing.Comment: 11 pages, 3 figure

Prediction and control of nonlinear dynamical systems using machine learning

Künstliche Intelligenz und Machine Learning erfreuen sich in Folge der rapide gestiegenen Rechenleistung immer größerer Popularität. Sei es autonomes Fahren, Gesichtserkennung, bildgebende Diagnostik in der Medizin oder Robotik – die Anwendungsvielfalt scheint keine Grenzen zu kennen. Um jedoch systematischen Bias und irreführende Ergebnisse zu vermeiden, ist ein tiefes Verständnis der Methoden und ihrer Sensitivitäten von Nöten. Anhand der Vorhersage chaotischer Systeme mit Reservoir Computing – einem künstlichen rekurrenten neuronalem Netzwerk – wird im Rahmen dieser Dissertation beleuchtet, wie sich verschiedene Eigenschaften des Netzwerks auf die Vorhersagekraft und Robustheit auswirken. Es wird gezeigt, wie sich die Variabilität der Vorhersagen – sowohl was die exakte zukünftige Trajektorie betrifft als auch das statistische Langzeitverhalten (das "Klima") des Systems – mit geeigneter Parameterwahl signifikant reduzieren lässt. Die Nichtlinearität der Aktivierungsfunktion spielt hierbei eine besondere Rolle, weshalb ein Skalierungsparameter eingeführt wird, um diese zu kontrollieren. Des Weiteren werden differenzielle Eigenschaften des Netzwerkes untersucht und gezeigt, wie ein kontrolliertes Entfernen der "richtigen" Knoten im Netzwerk zu besseren Vorhersagen führt und die Größe des Netzwerkes stark reduziert werden kann bei gleichzeitig nur moderater Verschlechterung der Ergebnisse. Dies ist für Hardware Realisierungen von Reservoir Computing wie zum Beispiel Neuromorphic Computing relevant, wo möglichst kleine Netzwerke von Vorteil sind. Zusätzlich werden unterschiedliche Netzwerktopologien wie Small World Netzwerke und skalenfreie Netzwerke beleuchtet. Mit den daraus gewonnenen Erkenntnissen für bessere Vorhersagen von nichtlinearen dynamischen Systemen wird eine neue Kontrollmethode entworfen, die es ermöglicht, dynamische Systeme flexibel in verschiedenste Zielzustände zu lenken. Hierfür wird – anders als bei vielen bisherigen Ansätzen – keine Kenntnis der zugrundeliegenden Gleichungen des Systems erfordert. Ebenso wird nur eine begrenzte Datenmenge verlangt, um Reservoir Computing hinreichend zu trainieren. Zudem ist es nicht nur möglich, chaotisches Verhalten in einen periodischen Zustand zu zwingen, sondern auch eine Kontrolle auf komplexere Zielzustände wie intermittentes Verhalten oder ein spezifischer anderer chaotischer Zustand. Dies ermöglicht eine Vielzahl neuer potenzieller realer Anwendungen, von personalisierten Herzschrittmachern bis hin zu Kontrollvorrichtungen für Raketentriebwerke zur Unterbindung von kritischen Verbrennungsinstabilitäten. Als Schritt zur Weiterentwicklung von Reservoir Computing zu einem verbesserten hybriden System, das nicht nur rein datenbasiert arbeitet, sondern auch physikalische Zusammenhänge berücksichtigt, wird ein Ansatz vorgestellt, um lineare und nichtlinearen Kausalitätsstrukturen zu separieren. Dies kann verwendet werden, um Systemgleichungen oder Restriktionen für ein hybrides System zur Vorhersage oder Kontrolle abzuleiten.Artificial intelligence and machine learning are becoming increasingly popular as a result of the rapid increase in computing power. Be it autonomous driving, facial recognition, medical imaging diagnostics or robotics – the variety of applications seems to have no limits. However, to avoid systematic bias and misleading results, a deep understanding of the methods and their sensitivities is needed. Based on the prediction of chaotic systems with reservoir computing – an artificial recurrent neural network – this dissertation sheds light on how different properties of the network affect the predictive power and robustness. It is shown how the variability of the predictions – both in terms of the exact short-term predictions and the long-term statistical behaviour (the "climate") of the system – can be significantly reduced with appropriate parameter choices. The nonlinearity of the activation function plays a special role here, thus a scaling parameter is introduced to control it. Furthermore, differential properties of the network are investigated and it is shown how a controlled removal of the right nodes in the network leads to better predictions, whereas the size of the network can be greatly reduced while only moderately degrading the results. This is relevant for hardware realizations of reservoir computing such as neuromorphic computing, where networks that are as small as possible are advantageous. Additionally, different network topologies such as small world networks and scale-free networks are investigated. With the insights gained for better predictions of nonlinear dynamical systems, a new control method is designed that allows dynamical systems to be flexibly forced into a wide variety of dynamical target states. For this – unlike many previous approaches – no knowledge of the underlying equations of the system is required. Further, only a limited amount of data is needed to sufficiently train reservoir computing. Moreover, it is possible not only to force chaotic behavior to a periodic state, but also to control for more complex target states such as intermittent behavior or a specific different chaotic state. This enables a variety of new potential real-world applications, from personalized cardiac pacemakers to control devices for rocket engines to suppress critical combustion instabilities. As a step toward advancing reservoir computing to an improved hybrid system that is not only purely data-based but also takes into account physical relationships, an approach is presented to separate linear and nonlinear causality structures. This can be used to derive system equations or constraints for a hybrid prediction or control system

Transport of quantum excitations coupled to spatially extended nonlinear many-body systems

The role of noise in the transport properties of quantum excitations is a topic of great importance in many fields, from organic semiconductors for technological applications to light-harvesting complexes in photosynthesis. In this paper we study a semi-classical model where a tight-binding Hamiltonian is fully coupled to an underlying spatially extended nonlinear chain of atoms. We show that the transport properties of a quantum excitation are subtly modulated by (i) the specific type (local vs non-local) of exciton-phonon coupling and by (ii) nonlinear effects of the underlying lattice. We report a non-monotonic dependence of the exciton diffusion coefficient on temperature, in agreement with earlier predictions, as a direct consequence of the lattice-induced fluctuations in the hopping rates due to long-wavelength vibrational modes. A standard measure of transport efficiency confirms that both nonlinearity in the underlying lattice and off-diagonal exciton-phonon coupling promote transport efficiency at high temperatures, preventing the Zeno-like quench observed in other models lacking an explicit noise-providing dynamical system

Theory and Practice of Computing with Excitable Dynamics

Reservoir computing (RC) is a promising paradigm for time series processing. In this paradigm, the desired output is computed by combining measurements of an excitable system that responds to time-dependent exogenous stimuli. The excitable system is called a reservoir and measurements of its state are combined using a readout layer to produce a target output. The power of RC is attributed to an emergent short-term memory in dynamical systems and has been analyzed mathematically for both linear and nonlinear dynamical systems. The theory of RC treats only the macroscopic properties of the reservoir, without reference to the underlying medium it is made of. As a result, RC is particularly attractive for building computational devices using emerging technologies whose structure is not exactly controllable, such as self-assembled nanoscale circuits. RC has lacked a formal framework for performance analysis and prediction that goes beyond memory properties. To provide such a framework, here a mathematical theory of memory and information processing in ordered and disordered linear dynamical systems is developed. This theory analyzes the optimal readout layer for a given task. The focus of the theory is a standard model of RC, the echo state network (ESN). An ESN consists of a fixed recurrent neural network that is driven by an external signal. The dynamics of the network is then combined linearly with readout weights to produce the desired output. The readout weights are calculated using linear regression. Using an analysis of regression equations, the readout weights can be calculated using only the statistical properties of the reservoir dynamics, the input signal, and the desired output. The readout layer weights can be calculated from a priori knowledge of the desired function to be computed and the weight matrix of the reservoir. This formulation explicitly depends on the input weights, the reservoir weights, and the statistics of the target function. This formulation is used to bound the expected error of the system for a given target function. The effects of input-output correlation and complex network structure in the reservoir on the computational performance of the system have been mathematically characterized. Far from the chaotic regime, ordered linear networks exhibit a homogeneous decay of memory in different dimensions, which keeps the input history coherent. As disorder is introduced in the structure of the network, memory decay becomes inhomogeneous along different dimensions causing decoherence in the input history, and degradation in task-solving performance. Close to the chaotic regime, the ordered systems show loss of temporal information in the input history, and therefore inability to solve tasks. However, by introducing disorder and therefore heterogeneous decay of memory the temporal information of input history is preserved and the task-solving performance is recovered. Thus for systems at the edge of chaos, disordered structure may enhance temporal information processing. Although the current framework only applies to linear systems, in principle it can be used to describe the properties of physical reservoir computing, e.g., photonic RC using short coherence-length light