Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory--nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behaviour in phase space characterised by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking

Input-to-State Representation in linear reservoirs dynamics

Reservoir computing is a popular approach to design recurrent neural networks, due to its training simplicity and approximation performance. The recurrent part of these networks is not trained (e.g., via gradient descent), making them appealing for analytical studies by a large community of researchers with backgrounds spanning from dynamical systems to neuroscience. However, even in the simple linear case, the working principle of these networks is not fully understood and their design is usually driven by heuristics. A novel analysis of the dynamics of such networks is proposed, which allows the investigator to express the state evolution using the controllability matrix. Such a matrix encodes salient characteristics of the network dynamics; in particular, its rank represents an input-indepedent measure of the memory capacity of the network. Using the proposed approach, it is possible to compare different reservoir architectures and explain why a cyclic topology achieves favourable results as verified by practitioners

Cerebellar atrophy in Parkinson's disease and its implication for network connectivity.

Pathophysiological and atrophic changes in the cerebellum are documented in Parkinson's disease. Without compensatory activity, such abnormalities could potentially have more widespread effects on both motor and non-motor symptoms. We examined how atrophic change in the cerebellum impacts functional connectivity patterns within the cerebellum and between cerebellar-cortical networks in 42 patients with Parkinson's disease and 29 control subjects. Voxel-based morphometry confirmed grey matter loss across the motor and cognitive cerebellar territories in the patient cohort. The extent of cerebellar atrophy correlated with decreased resting-state connectivity between the cerebellum and large-scale cortical networks, including the sensorimotor, dorsal attention and default networks, but with increased connectivity between the cerebellum and frontoparietal networks. The severity of patients' motor impairment was predicted by a combination of cerebellar atrophy and decreased cerebellar-sensorimotor connectivity. These findings demonstrate that cerebellar atrophy is related to both increases and decreases in cerebellar-cortical connectivity in Parkinson's disease, identifying potential cerebellar driven functional changes associated with sensorimotor deficits. A post hoc analysis exploring the effect of atrophy in the subthalamic nucleus, a cerebellar input source, confirmed that a significant negative relationship between grey matter volume and intrinsic cerebellar connectivity seen in controls was absent in the patients. This suggests that the modulatory relationship of the subthalamic nucleus on intracerebellar connectivity is lost in Parkinson's disease, which may contribute to pathological activation within the cerebellum. The results confirm significant changes in cerebellar network activity in Parkinson's disease and reveal that such changes occur in association with atrophy of the cerebellum

Reducing network size and improving prediction stability of reservoir computing

Reservoir computing is a very promising approach for the prediction of complex nonlinear dynamical systems. Besides capturing the exact short-term trajectories of nonlinear systems, it has also proved to reproduce its characteristic long-term properties very accurately. However, predictions do not always work equivalently well. It has been shown that both short- and long-term predictions vary significantly among different random realizations of the reservoir. In order to gain an understanding on when reservoir computing works best, we investigate some differential properties of the respective realization of the reservoir in a systematic way. We find that removing nodes that correspond to the largest weights in the output regression matrix reduces outliers and improves overall prediction quality. Moreover, this allows to effectively reduce the network size and, therefore, increase computational efficiency. In addition, we use a nonlinear scaling factor in the hyperbolic tangent of the activation function. This adjusts the response of the activation function to the range of values of the input variables of the nodes. As a consequence, this reduces the number of outliers significantly and increases both the short- and long-term prediction quality for the nonlinear systems investigated in this study. Our results demonstrate that a large optimization potential lies in the systematical refinement of the differential reservoir properties for a given dataset.Comment: 11 pages, 8 figures, published in Chao

Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

We present the results of the comparative performance-versus-complexity analysis for the several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison is carried out using an experimental set-up with the transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN-based equalizers: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. First, we present the general expressions for the computational complexity associated with each NN type; these are given in terms of real multiplications per symbol. We demonstrate that in the experimental system considered, the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the largest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). Then, we examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to some lower levels, the three-layer perceptron provides the best performance

On the neural basis of emotion processing in depression and anxiety : an fMRI study in outpatients

The wide implications of emotions in our social and private life, and the far-reaching consequences of dysfunctions in emotional processing, have made emotion one of the most widely studied psychological processes. During the last decades, there have been numerous attempts to formulate neurobiological and cognitive theories of emotion. Mental disorders, e.g., schizophrenia, bipolar disorders, mood disorder, anxiety, are associated with dysfunction of emotional processing. These dysfunctions may be caused by abnormalities at neural level. From all psychiatric disorders characterized by emotional disturbance, major depressive disorder and anxiety disorders are the most prevalent in our society. For outcome improvement, a clear delineation of the neural mechanism of emotional processing in community-based outpatients is of fundamental importance to understanding their underlying mechanisms. We use the functional magnetic resonance imaging (fMRI) method for studying different cognitive and emotional functions in major depressive disorder and anxiety disorders. The findings presented herein indicate that dysfunctions in the neural circuitry of emotional processing are different in depression and anxiety. Furthermore we find that comorbidity of depression and anxiety cannot be regarded as a summation of the two. We also show that even if there are no gross abnormalities at the neural level, abnormalities in the neural network may cause dysfunctions of emotional processes in mild-remitted patients and participants with high vulnerability for affective disorders. This finding unveils a much more complex picture of emotion perception than the present day theories account for. De brede implicaties van emoties in ons sociale en privéleven, en de verstrekkende gevolgen van problemen in de verwerking van emoties, hebben emotie één van de meest bestudeerde psychologische processen gemaakt. De laatste decennia zijn talrijke pogingen gedaan om neurobiologische en cognitieve theorieën van emotie te formuleren. Mentale stoornissen, zoals schizofrenie, en bipolaire, stemmings-, en angststoornissen, zijn geassocieerd met problemen in de verwerking van emoties. Deze disfuncties zouden op neuronaal niveau veroorzaakt kunnen worden. De meest voorkomende psychiatrische stoornissen in onze samenleving zijn depressie en angststoornissen. Voor het verbeteren van het behandelresultaat, is het van fundamenteel belang om inzicht te krijgen in de neurale mechanismen, die betrokken zijn bij de verwerking van emoties in poliklinische patiënten uit de gemeenschap. We gebruiken de methode van functionele magnetische resonantie (fMRI) om verschillende cognitieve en emotionele functies te onderzoeken in depressie en angststoornissen. De bevindingen, die hier worden gepresenteerd, geven aan dat disfuncties in het neurale circuit van emotieverwerking verschillend zijn in depressie en angst. Verder, vinden we dat comorbiditeit van depressie en angst niet kan worden opgevat als een simpele opsomming van de twee. Ook laten we zien dat, ondanks er geen grote verschillen met gezonde personen aanwezig zijn op neuronaal niveau, abnormaliteiten in het neurale netwerk disfuncties kunnen veroorzaken in de emotionele verwerking van licht verbeterde patiënten en personen met een hoge kwetsbaarheid voor affectieve stoornissen. Deze bevinding onthult een gecompliceerder beeld van de perceptie van emotie dan huidige theorieën aangeven.