Efficient Cross-Validation of Echo State Networks

Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cross-validating ESNs and introduce an efficient algorithm for implementing them. The component that dominates the time complexity of the already quite fast ESN training remains constant (does not scale up with $k$) in our proposed method of doing $k$-fold cross-validation. The component that does scale linearly with $k$ starts dominating only in some not very common situations. Thus in many situations $k$-fold cross-validation of ESNs can be done for virtually the same time complexity as a simple single split validation. Space complexity can also remain the same. We also discuss when the proposed validation schemes for ESNs could be beneficial and empirically investigate them on several different real-world datasets.Comment: Accepted in ICANN'19 Workshop on Reservoir Computin

Ensemble reservoir computing for dynamical systems: prediction of phase-space stable region for hadron storage rings

We investigate the ability of an ensemble reservoir computing approach to predict the long-term behaviour of the phase-space region in which the motion of charged particles in hadron storage rings is bounded, the so-called dynamic aperture. Currently, the calculation of the phase-space stability region of hadron storage rings is performed through direct computer simulations, which are resource- and time-intensive processes. Echo State Networks (ESN) are a class of recurrent neural networks that are computationally effective, since they avoid backpropagation and require only cross-validation. Furthermore, they have been proven to be universal approximants of dynamical systems. In this paper, we present the performance reached by ESN based on an ensemble approach for the prediction of the phase-space stability region and compare it with analytical scaling laws based on the stability-time estimate of the Nekhoroshev theorem for Hamiltonian systems. We observe that the proposed ESN approach is capable of effectively predicting the time evolution of the extent of the dynamic aperture, improving the predictions by analytical scaling laws, thus providing an efficient surrogate model.We investigate the ability of an ensemble reservoir computing approach to predict the long-term behaviour of the phase-space region in which the motion of charged particles in hadron storage rings is bounded, the so-called dynamic aperture. Currently, the calculation of the phase-space stability region of hadron storage rings is performed through direct computer simulations, which are resource- and time-intensive processes. Echo State Networks (ESN) are a class of recurrent neural networks that are computationally effective, since they avoid backpropagation and require only cross-validation. Furthermore, they have been proven to be universal approximants of dynamical systems. In this paper, we present the performance reached by ESN based on an ensemble approach for the prediction of the phase-space stability region and compare it with analytical scaling laws based on the stability-time estimate of the Nekhoroshev theorem for Hamiltonian systems. We observe that the proposed ESN approach is capable of effectively predicting the time evolution of the extent of the dynamic aperture, improving the predictions by analytical scaling laws, thus providing an efficient surrogate model

Efficient Optimization of Echo State Networks for Time Series Datasets

Echo State Networks (ESNs) are recurrent neural networks that only train their output layer, thereby precluding the need to backpropagate gradients through time, which leads to significant computational gains. Nevertheless, a common issue in ESNs is determining its hyperparameters, which are crucial in instantiating a well performing reservoir, but are often set manually or using heuristics. In this work we optimize the ESN hyperparameters using Bayesian optimization which, given a limited budget of function evaluations, outperforms a grid search strategy. In the context of large volumes of time series data, such as light curves in the field of astronomy, we can further reduce the optimization cost of ESNs. In particular, we wish to avoid tuning hyperparameters per individual time series as this is costly; instead, we want to find ESNs with hyperparameters that perform well not just on individual time series but rather on groups of similar time series without sacrificing predictive performance significantly. This naturally leads to a notion of clusters, where each cluster is represented by an ESN tuned to model a group of time series of similar temporal behavior. We demonstrate this approach both on synthetic datasets and real world light curves from the MACHO survey. We show that our approach results in a significant reduction in the number of ESN models required to model a whole dataset, while retaining predictive performance for the series in each cluster

A novel time series analysis approach for prediction of dialysis in critically ill patients using echo-state networks

Background: Echo-state networks (ESN) are part of a group of reservoir computing methods and are basically a form of recurrent artificial neural networks (ANN). These methods can perform classification tasks on time series data. The recurrent ANN of an echo-state network has an 'echo-state' characteristic. This 'echo-state' functions as a fading memory: samples that have been introduced into the network in a further past, are faded away. The echostate approach for the training of recurrent neural networks was first described by Jaeger H. et al. In clinical medicine, until this moment, no original research articles have been published to examine the use of echo-state networks. Methods: This study examines the possibility of using an echo-state network for prediction of dialysis in the ICU. Therefore, diuresis values and creatinine levels of the first three days after ICU admission were collected from 830 patients admitted to the intensive care unit (ICU) between May 31th 2003 and November 17th 2007. The outcome parameter was the performance by the echo-state network in predicting the need for dialysis between day 5 and day 10 of ICU admission. Patients with an ICU length of stay < 10 days or patients that received dialysis in the first five days of ICU admission were excluded. Performance by the echo-state network was then compared by means of the area under the receiver operating characteristic curve (AUC) with results obtained by two other time series analysis methods by means of a support vector machine (SVM) and a naive Bayes algorithm (NB). Results: The AUC's in the three developed echo-state networks were 0.822, 0.818, and 0.817. These results were comparable to the results obtained by the SVM and the NB algorithm. Conclusions: This proof of concept study is the first to evaluate the performance of echo-state networks in an ICU environment. This echo-state network predicted the need for dialysis in ICU patients. The AUC's of the echo-state networks were good and comparable to the performance of other classification algorithms. Moreover, the echostate network was more easily configured than other time series modeling technologies

Training Echo State Networks with Regularization through Dimensionality Reduction

In this paper we introduce a new framework to train an Echo State Network to predict real valued time-series. The method consists in projecting the output of the internal layer of the network on a space with lower dimensionality, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework. Finally, we provide an insight on the effectiveness of the implemented mechanics through a visualization of the trajectory in the phase space and relying on the methodologies of nonlinear time-series analysis. By applying our method on well known chaotic systems, we provide evidence that the lower dimensional embedding retains the dynamical properties of the underlying system better than the full-dimensional internal states of the network