Comparison of echo state network output layer classification methods on noisy data

Echo state networks are a recently developed type of recurrent neural network where the internal layer is fixed with random weights, and only the output layer is trained on specific data. Echo state networks are increasingly being used to process spatiotemporal data in real-world settings, including speech recognition, event detection, and robot control. A strength of echo state networks is the simple method used to train the output layer - typically a collection of linear readout weights found using a least squares approach. Although straightforward to train and having a low computational cost to use, this method may not yield acceptable accuracy performance on noisy data. This study compares the performance of three echo state network output layer methods to perform classification on noisy data: using trained linear weights, using sparse trained linear weights, and using trained low-rank approximations of reservoir states. The methods are investigated experimentally on both synthetic and natural datasets. The experiments suggest that using regularized least squares to train linear output weights is superior on data with low noise, but using the low-rank approximations may significantly improve accuracy on datasets contaminated with higher noise levels.Comment: 8 pages. International Joint Conference on Neural Networks (IJCNN 2017

Modeling neural plasticity in echo state networks for time series prediction

In this paper, we investigate the influence of neural plasticity on the learning performance of echo state networks (ESNs) and supervised learning algorithms in training readout connections for two time series prediction problems including the sunspot time series and the Mackey Glass chaotic system. We implement two different plasticity rules that are expected to improve the prediction performance, namely, anti-Oja learning rule and the Bienenstock-Cooper-Munro (BCM) learning rule combined with both offline and online learning of the readout connections. Our experimental results have demonstrated that the neural plasticity can more significantly enhance the learning in offline learning than in online learning

On the importance of sluggish state memory for learning long term dependency

The vanishing gradients problem inherent in Simple Recurrent Networks (SRN) trained with back-propagation, has led to a significant shift towards the use of Long Short-term Memory (LSTM) and Echo State Networks (ESN), which overcome this problem through either second order error-carousel schemes or different learning algorithms respectively. This paper re-opens the case for SRN-based approaches, by considering a variant, the Multi-recurrent Network (MRN). We show that memory units embedded within its architecture can ameliorate against the vanishing gradient problem, by providing variable sensitivity to recent and more historic information through layer- and self-recurrent links with varied weights, to form a so-called sluggish state-based memory. We demonstrate that an MRN, optimised with noise injection, is able to learn the long term dependency within a complex grammar induction task, significantly outperforming the SRN, NARX and ESN. Analysis of the internal representations of the networks, reveals that sluggish state-based representations of the MRN are best able to latch on to critical temporal dependencies spanning variable time delays, to maintain distinct and stable representations of all underlying grammar states. Surprisingly, the ESN was unable to fully learn the dependency problem, suggesting the major shift towards this class of models may be premature

Reservoir computing approaches for representation and classification of multivariate time series

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir Computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this paper we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared to other RC methods, our model space yields better representations and attains comparable computational performance, thanks to an intermediate dimensionality reduction procedure. As a second contribution we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared to other MTS classifiers, including deep learning models and time series kernels. Results obtained on benchmark and real-world MTS datasets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy

Dynamical systems as temporal feature spaces

Parameterized state space models in the form of recurrent networks are often used in machine learning to learn from data streams exhibiting temporal dependencies. To break the black box nature of such models it is important to understand the dynamical features of the input driving time series that are formed in the state space. We propose a framework for rigorous analysis of such state representations in vanishing memory state space models such as echo state networks (ESN). In particular, we consider the state space a temporal feature space and the readout mapping from the state space a kernel machine operating in that feature space. We show that: (1) The usual ESN strategy of randomly generating input-to-state, as well as state coupling leads to shallow memory time series representations, corresponding to cross-correlation operator with fast exponentially decaying coefficients; (2) Imposing symmetry on dynamic coupling yields a constrained dynamic kernel matching the input time series with straightforward exponentially decaying motifs or exponentially decaying motifs of the highest frequency; (3) Simple cycle high-dimensional reservoir topology specified only through two free parameters can implement deep memory dynamic kernels with a rich variety of matching motifs. We quantify richness of feature representations imposed by dynamic kernels and demonstrate that for dynamic kernel associated with cycle reservoir topology, the kernel richness undergoes a phase transition close to the edge of stability.Comment: 45 pages, 17 figures, accepte

Investigating the Predictability of a Chaotic Time-Series Data using Reservoir Computing, Deep-Learning and Machine- Learning on the Short-, Medium- and Long-Term Pricing of Bitcoin and Ethereum.

This study will investigate the predictability of a Chaotic time-series data using Reservoir computing (Echo State Network), Deep-Learning(LSTM) and Machine- Learning(Linear, Bayesian, ElasticNetCV , Random Forest, XGBoost Regression and a machine learning Neural Network) on the short (1-day out prediction), medium (5-day out prediction) and long-term (30-day out prediction) pricing of Bitcoin and Ethereum Using a range of machine learning tools, to perform feature selection by permutation importance to select technical indicators on the individual cryptocurrencies, to ensure the datasets are the best for predictions per cryptocurrency while reducing noise within the models. The predictability of these two chaotic time-series is then compared to evaluate the models to find the best fit model. The models are fine-tuned, with hyperparameters, design of the network within the LSTM and the reservoir size within the Echo State Network being adjusted to improve accuracy and speed. This research highlights the effect of the trends within the cryptocurrency and its effect on predictive models, these models will then be optimized with hyperparameter tuning, and be evaluated to compare the models across the two currencies. It is found that the datasets for each cryptocurrency are different, due to the different permutation importance, which does not affect the overall predictability of the models with the short and medium-term predictions having the same models being the top performers. This research confirms that the chaotic data although can have positive results for shortand medium-term prediction, for long-term prediction, technical analysis basedprediction is not sufficient

Modellazione della ricarica degli acquiferi carbonatici dell’Appennino meridionale (Italia), a scala regionale e di bacino

In molti Paesi del mondo, gli acquiferi carbonatici sono la principale fonte di approvvigionamento di acque sotterranee poiché le loro risorse idriche sotterranee sono cruciali per lo sviluppo socio-economico del territorio e la conservazione degli ecosistemi acquatici (Goldscheider, 2012), terrestri e costieri che dipendono dalle acque sotterranee (Groundwater Dependent Ecosystems, GDE) (Foster et al., 2015). Per questi acquiferi, la stima della ricarica e la previsione dei processi “afflusso-deflusso”, a diverse scale spazio-temporali e con metodologie sempre più innovative, è un fondamentale presupposto per un uso sostenibile delle acque sotterranee, una corretta gestione dei sistemi di approvvigionamento-adduzione-distribuzione della risorsa idrica ed un’efficace prevenzione dei rischi legati ai cambiamenti climatici in atto, a scala globale e locale. Nell’Appennino meridionale la misura diretta delle diverse grandezze idrologiche, necessarie per la stima della ricarica degli acquiferi carbonatici, attraverso il bilancio di massa, è molto complessa e impegnativa, data la mancanza, ad alta quota, di stazioni pluvio-termometriche (Allocca et al., 2014), lisimetriche e idrometriche per la misura diretta della pioggia, temperatura dell’aria, dell’evapotraspirazione e del deflusso idrico superficiale. Pertanto, in tale contesto, l’integrazione di dati idrologici terrestri e telerilevati può rappresentare un valido approccio per superare e/o limitare le incertezze legate alla mancanza di misure e serie storiche di campo. L’obiettivo della presente tesi di dottorato è stato la modellazione della ricarica, degli acquiferi carbonatici di un’ampia porzione dell’Appennino meridionale, dalla scala regionale e medio-annua alla scala di bacino e giornaliera. La modellazione della ricarica, a scala regionale e medio-annua, ha riguardato 40 acquiferi carbonatici di una vasta area dell’Appennino meridionale, estesa circa 19.339 km2 e ricadente nelle regioni Abruzzo, Lazio, Molise, Campania e Basilicata. La modellazione è stata effettuata attraverso l’integrazione di dati idrologici terrestri (temperatura dell’aria e precipitazioni) con dati telerilevati dal satellite MODIS, per la stima dell’evapotraspirazione reale (ETa). Per il periodo 2000-2014, le serie storiche idrologiche di pioggia di 266 pluviometri e di temperatura dell’aria di 150 stazioni termometriche, sono state utilizzate per la ricostruzione dei modelli regionali distribuiti di precipitazione, temperatura dell’aria e deflusso idrico globale. La stima della ricarica con dati MODIS è stata, poi, confrontata con le stime derivanti da altre metodologie empiriche (Coutagne, 1954; Turc, 1954; Thornthwaite, 1948), largamente validate per l’Appennino carbonatico meridionale. La modellazione della ricarica alla scala di bacino e giornaliera ha riguardato l’acquifero carbonatico del Monte Cervialto (Campania) ed è stata eseguita attraverso la simulazione dei processi “afflusso-deflusso” per il bacino sotterraneo di alimentazione della sorgente Sanità di Caposele (AV), mediante l’utilizzo di 3 differenti modelli di reti neurali artificiali: Multilayer Perceptron (MLP), Long Short Term Memory (LSTM) ed Echo State Network (ESN). Sono state effettuate previsioni per diversi archi temporali: a 60 giorni, a 120 giorni ed a 180 giorni. I risultati evidenziano che, a scala regionale e medio-annua, per tutti i 40 acquiferi carbonatici dell’Appennino meridionale la ricarica calcolata con i dati di ETa MODIS, è pari a 448 mm, corrispondente ad un volume medio-annuo di 3.834 × 106 m3. Considerando i valori dell’ETa stimati con la formula di Coutagne (1954), Turc (1954) e Thornthwaite (1948), la ricarica medio-annua è risultata pari, rispettivamente, a 533, 494 e 437 mm, corrispondente ad un volume medio-annuo rispettivamente di 4.561×106 m3, 4.228×106 m3 e 3.740×106 m3. A scala di bacino e giornaliera, per il bacino sotterraneo di alimentazione della sorgente Sanità di Caposele, la simulazione dei processi “afflusso-deflusso” mostra che i tre modelli di reti neurali artificiali a 60 giorni forniscono un’ottima risposta previsionale, coerente con l’andamento reale dell’idrogramma sorgivo, replicando i punti nodali, di massimo e di minimo deflusso sorgivo, e le diverse dinamiche idrologiche di ricarica e recessione, con errori di previsione mediamente del 5%. In conclusione, sulla base del primo approccio, la modellazione a scala regionale e medio-annua della ricarica con l’uso integrato di dati idrologici terrestri con dati satellitari MODIS apre nuove prospettive per la stima dei volumi delle risorse idriche sotterranee, consentendo di limitare le incertezze derivanti dalla totale assenza di copertura meteorologica delle aree di ricarica degli acquiferi carbonatici dell’Appennino meridionale. Inoltre, l’applicazione delle reti neurali artificiali in campo idrogeologico si è rivelato uno strumento innovativo per la modellazione, a scala di bacino e giornaliera, di processi non lineari, quali “ricarica-deflusso” in sistemi discontinui, eterogenei ed anisotropi come gli acquiferi carbonatici dell’Appennino meridionale

Reservoir Computing: computation with dynamical systems

In het onderzoeksgebied Machine Learning worden systemen onderzocht die kunnen leren op basis van voorbeelden. Binnen dit onderzoeksgebied zijn de recurrente neurale netwerken een belangrijke deelgroep. Deze netwerken zijn abstracte modellen van de werking van delen van de hersenen. Zij zijn in staat om zeer complexe temporele problemen op te lossen maar zijn over het algemeen zeer moeilijk om te trainen. Recentelijk zijn een aantal gelijkaardige methodes voorgesteld die dit trainingsprobleem elimineren. Deze methodes worden aangeduid met de naam Reservoir Computing. Reservoir Computing combineert de indrukwekkende rekenkracht van recurrente neurale netwerken met een eenvoudige trainingsmethode. Bovendien blijkt dat deze trainingsmethoden niet beperkt zijn tot neurale netwerken, maar kunnen toegepast worden op generieke dynamische systemen. Waarom deze systemen goed werken en welke eigenschappen bepalend zijn voor de prestatie is evenwel nog niet duidelijk. Voor dit proefschrift is onderzoek gedaan naar de dynamische eigenschappen van generieke Reservoir Computing systemen. Zo is experimenteel aangetoond dat de idee van Reservoir Computing ook toepasbaar is op niet-neurale netwerken van dynamische knopen. Verder is een maat voorgesteld die gebruikt kan worden om het dynamisch regime van een reservoir te meten. Tenslotte is een adaptatieregel geïntroduceerd die voor een breed scala reservoirtypes de dynamica van het reservoir kan afregelen tot het gewenste dynamisch regime. De technieken beschreven in dit proefschrift zijn gedemonstreerd op verschillende academische en ingenieurstoepassingen