Sparsity in Reservoir Computing Neural Networks

Reservoir Computing (RC) is a well-known strategy for designing Recurrent Neural Networks featured by striking efficiency of training. The crucial aspect of RC is to properly instantiate the hidden recurrent layer that serves as dynamical memory to the system. In this respect, the common recipe is to create a pool of randomly and sparsely connected recurrent neurons. While the aspect of sparsity in the design of RC systems has been debated in the literature, it is nowadays understood mainly as a way to enhance the efficiency of computation, exploiting sparse matrix operations. In this paper, we empirically investigate the role of sparsity in RC network design under the perspective of the richness of the developed temporal representations. We analyze both sparsity in the recurrent connections, and in the connections from the input to the reservoir. Our results point out that sparsity, in particular in input-reservoir connections, has a major role in developing internal temporal representations that have a longer short-term memory of past inputs and a higher dimension.Comment: This paper is currently under revie

Reservoir Topology in Deep Echo State Networks

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.Comment: Preprint of the paper published in the proceedings of ICANN 201

Biological neurons act as generalization filters in reservoir computing

Reservoir computing is a machine learning paradigm that transforms the transient dynamics of high-dimensional nonlinear systems for processing time-series data. Although reservoir computing was initially proposed to model information processing in the mammalian cortex, it remains unclear how the non-random network architecture, such as the modular architecture, in the cortex integrates with the biophysics of living neurons to characterize the function of biological neuronal networks (BNNs). Here, we used optogenetics and fluorescent calcium imaging to record the multicellular responses of cultured BNNs and employed the reservoir computing framework to decode their computational capabilities. Micropatterned substrates were used to embed the modular architecture in the BNNs. We first show that modular BNNs can be used to classify static input patterns with a linear decoder and that the modularity of the BNNs positively correlates with the classification accuracy. We then used a timer task to verify that BNNs possess a short-term memory of ~1 s and finally show that this property can be exploited for spoken digit classification. Interestingly, BNN-based reservoirs allow transfer learning, wherein a network trained on one dataset can be used to classify separate datasets of the same category. Such classification was not possible when the input patterns were directly decoded by a linear decoder, suggesting that BNNs act as a generalization filter to improve reservoir computing performance. Our findings pave the way toward a mechanistic understanding of information processing within BNNs and, simultaneously, build future expectations toward the realization of physical reservoir computing systems based on BNNs.Comment: 31 pages, 5 figures, 3 supplementary figure

Ensemble reservoir computing for dynamical systems: prediction of phase-space stable region for hadron storage rings

We investigate the ability of an ensemble reservoir computing approach to predict the long-term behaviour of the phase-space region in which the motion of charged particles in hadron storage rings is bounded, the so-called dynamic aperture. Currently, the calculation of the phase-space stability region of hadron storage rings is performed through direct computer simulations, which are resource- and time-intensive processes. Echo State Networks (ESN) are a class of recurrent neural networks that are computationally effective, since they avoid backpropagation and require only cross-validation. Furthermore, they have been proven to be universal approximants of dynamical systems. In this paper, we present the performance reached by ESN based on an ensemble approach for the prediction of the phase-space stability region and compare it with analytical scaling laws based on the stability-time estimate of the Nekhoroshev theorem for Hamiltonian systems. We observe that the proposed ESN approach is capable of effectively predicting the time evolution of the extent of the dynamic aperture, improving the predictions by analytical scaling laws, thus providing an efficient surrogate model.We investigate the ability of an ensemble reservoir computing approach to predict the long-term behaviour of the phase-space region in which the motion of charged particles in hadron storage rings is bounded, the so-called dynamic aperture. Currently, the calculation of the phase-space stability region of hadron storage rings is performed through direct computer simulations, which are resource- and time-intensive processes. Echo State Networks (ESN) are a class of recurrent neural networks that are computationally effective, since they avoid backpropagation and require only cross-validation. Furthermore, they have been proven to be universal approximants of dynamical systems. In this paper, we present the performance reached by ESN based on an ensemble approach for the prediction of the phase-space stability region and compare it with analytical scaling laws based on the stability-time estimate of the Nekhoroshev theorem for Hamiltonian systems. We observe that the proposed ESN approach is capable of effectively predicting the time evolution of the extent of the dynamic aperture, improving the predictions by analytical scaling laws, thus providing an efficient surrogate model

The trend of disruption in the functional brain network topology of Alzheimer’s disease

Alzheimer’s disease (AD) is a progressive disorder associated with cognitive dysfunction that alters the brain’s functional connectivity. Assessing these alterations has become a topic of increasing interest. However, a few studies have examined different stages of AD from a complex network perspective that cover different topological scales. This study used resting state fMRI data to analyze the trend of functional connectivity alterations from a cognitively normal (CN) state through early and late mild cognitive impairment (EMCI and LMCI) and to Alzheimer’s disease. The analyses had been done at the local (hubs and activated links and areas), meso (clustering, assortativity, and rich-club), and global (small-world, small-worldness, and efficiency) topological scales. The results showed that the trends of changes in the topological architecture of the functional brain network were not entirely proportional to the AD progression. There were network characteristics that have changed non-linearly regarding the disease progression, especially at the earliest stage of the disease, i.e., EMCI. Further, it has been indicated that the diseased groups engaged somatomotor, frontoparietal, and default mode modules compared to the CN group. The diseased groups also shifted the functional network towards more random architecture. In the end, the methods introduced in this paper enable us to gain an extensive understanding of the pathological changes of the AD process

The Trend of Disruption in the Functional Brain Network Topology of Alzheimer’s Disease

Alzheimer’s disease (AD) is a progressive disorder associated with cognitive dysfunction that alters the brain’s functional connectivity. Assessing these alterations has become a topic of increasing interest. However, a few studies have examined different stages of AD from a complex network perspective that cover different topological scales. This study used resting state fMRI data to analyze the trend of functional connectivity alterations from a cognitively normal (CN) state through early and late mild cognitive impairment (EMCI and LMCI) and to Alzheimer’s disease. The analyses had been done at the local (hubs and activated links and areas), meso (clustering, assortativity, and rich-club), and global (small-world, small-worldness, and efficiency) topological scales. The results showed that the trends of changes in the topological architecture of the functional brain network were not entirely proportional to the AD progression. There were network characteristics that have changed non-linearly regarding the disease progression, especially at the earliest stage of the disease, i.e., EMCI. Further, it has been indicated that the diseased groups engaged somatomotor, frontoparietal, and default mode modules compared to the CN group. The diseased groups also shifted the functional network towards more random architecture. In the end, the methods introduced in this paper enable us to gain an extensive understanding of the pathological changes of the AD process

Machine learning methods for the characterization and classification of complex data

This thesis work presents novel methods for the analysis and classification of medical images and, more generally, complex data. First, an unsupervised machine learning method is proposed to order anterior chamber OCT (Optical Coherence Tomography) images according to a patient's risk of developing angle-closure glaucoma. In a second study, two outlier finding techniques are proposed to improve the results of above mentioned machine learning algorithm, we also show that they are applicable to a wide variety of data, including fraud detection in credit card transactions. In a third study, the topology of the vascular network of the retina, considering it a complex tree-like network is analyzed and we show that structural differences reveal the presence of glaucoma and diabetic retinopathy. In a fourth study we use a model of a laser with optical injection that presents extreme events in its intensity time-series to evaluate machine learning methods to forecast such extreme events.El presente trabajo de tesis desarrolla nuevos métodos para el análisis y clasificación de imágenes médicas y datos complejos en general. Primero, proponemos un método de aprendizaje automático sin supervisión que ordena imágenes OCT (tomografía de coherencia óptica) de la cámara anterior del ojo en función del grado de riesgo del paciente de padecer glaucoma de ángulo cerrado. Luego, desarrollamos dos métodos de detección automática de anomalías que utilizamos para mejorar los resultados del algoritmo anterior, pero que su aplicabilidad va mucho más allá, siendo útil, incluso, para la detección automática de fraudes en transacciones de tarjetas de crédito. Mostramos también, cómo al analizar la topología de la red vascular de la retina considerándola una red compleja, podemos detectar la presencia de glaucoma y de retinopatía diabética a través de diferencias estructurales. Estudiamos también un modelo de un láser con inyección óptica que presenta eventos extremos en la serie temporal de intensidad para evaluar diferentes métodos de aprendizaje automático para predecir dichos eventos extremos.Aquesta tesi desenvolupa nous mètodes per a l’anàlisi i la classificació d’imatges mèdiques i dades complexes. Hem proposat, primer, un mètode d’aprenentatge automàtic sense supervisió que ordena imatges OCT (tomografia de coherència òptica) de la cambra anterior de l’ull en funció del grau de risc del pacient de patir glaucoma d’angle tancat. Després, hem desenvolupat dos mètodes de detecció automàtica d’anomalies que hem utilitzat per millorar els resultats de l’algoritme anterior, però que la seva aplicabilitat va molt més enllà, sent útil, fins i tot, per a la detecció automàtica de fraus en transaccions de targetes de crèdit. Mostrem també, com en analitzar la topologia de la xarxa vascular de la retina considerant-la una xarxa complexa, podem detectar la presència de glaucoma i de retinopatia diabètica a través de diferències estructurals. Finalment, hem estudiat un làser amb injecció òptica, el qual presenta esdeveniments extrems en la sèrie temporal d’intensitat. Hem avaluat diferents mètodes per tal de predir-los.Postprint (published version