Bidirectional deep-readout echo state networks

We propose a deep architecture for the classification of multivariate time series. By means of a recurrent and untrained reservoir we generate a vectorial representation that embeds temporal relationships in the data. To improve the memorization capability, we implement a bidirectional reservoir, whose last state captures also past dependencies in the input. We apply dimensionality reduction to the final reservoir states to obtain compressed fixed size representations of the time series. These are subsequently fed into a deep feedforward network trained to perform the final classification. We test our architecture on benchmark datasets and on a real-world use-case of blood samples classification. Results show that our method performs better than a standard echo state network and, at the same time, achieves results comparable to a fully-trained recurrent network, but with a faster training

Deep Randomized Neural Networks

Randomized Neural Networks explore the behavior of neural systems where the majority of connections are fixed, either in a stochastic or a deterministic fashion. Typical examples of such systems consist of multi-layered neural network architectures where the connections to the hidden layer(s) are left untrained after initialization. Limiting the training algorithms to operate on a reduced set of weights inherently characterizes the class of Randomized Neural Networks with a number of intriguing features. Among them, the extreme efficiency of the resulting learning processes is undoubtedly a striking advantage with respect to fully trained architectures. Besides, despite the involved simplifications, randomized neural systems possess remarkable properties both in practice, achieving state-of-the-art results in multiple domains, and theoretically, allowing to analyze intrinsic properties of neural architectures (e.g. before training of the hidden layers’ connections). In recent years, the study of Randomized Neural Networks has been extended towards deep architectures, opening new research directions to the design of effective yet extremely efficient deep learning models in vectorial as well as in more complex data domains. This chapter surveys all the major aspects regarding the design and analysis of Randomized Neural Networks, and some of the key results with respect to their approximation capabilities. In particular, we first introduce the fundamentals of randomized neural models in the context of feed-forward networks (i.e., Random Vector Functional Link and equivalent models) and convolutional filters, before moving to the case of recurrent systems (i.e., Reservoir Computing networks). For both, we focus specifically on recent results in the domain of deep randomized systems, and (for recurrent models) their application to structured domains

Deep Randomized Neural Networks

Randomized Neural Networks explore the behavior of neural systems where the majority of connections are fixed, either in a stochastic or a deterministic fashion. Typical examples of such systems consist of multi-layered neural network architectures where the connections to the hidden layer(s) are left untrained after initialization. Limiting the training algorithms to operate on a reduced set of weights inherently characterizes the class of Randomized Neural Networks with a number of intriguing features. Among them, the extreme efficiency of the resulting learning processes is undoubtedly a striking advantage with respect to fully trained architectures. Besides, despite the involved simplifications, randomized neural systems possess remarkable properties both in practice, achieving state-of-the-art results in multiple domains, and theoretically, allowing to analyze intrinsic properties of neural architectures (e.g. before training of the hidden layers' connections). In recent years, the study of Randomized Neural Networks has been extended towards deep architectures, opening new research directions to the design of effective yet extremely efficient deep learning models in vectorial as well as in more complex data domains. This chapter surveys all the major aspects regarding the design and analysis of Randomized Neural Networks, and some of the key results with respect to their approximation capabilities. In particular, we first introduce the fundamentals of randomized neural models in the context of feed-forward networks (i.e., Random Vector Functional Link and equivalent models) and convolutional filters, before moving to the case of recurrent systems (i.e., Reservoir Computing networks). For both, we focus specifically on recent results in the domain of deep randomized systems, and (for recurrent models) their application to structured domains

Reservoir computing approaches for representation and classification of multivariate time series

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir Computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this paper we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared to other RC methods, our model space yields better representations and attains comparable computational performance, thanks to an intermediate dimensionality reduction procedure. As a second contribution we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared to other MTS classifiers, including deep learning models and time series kernels. Results obtained on benchmark and real-world MTS datasets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy

Enhancing Signal Recognition Accuracy in Delay-Based Optical Reservoir Computing: A Comparative Analysis of Training Algorithms

To improve the accuracy of signal recognition in delay-based optical reservoir computing (RC) systems, this paper proposes the use of nonlinear algorithms at the output layer to replace traditional linear algorithms for training and testing datasets and apply them to the identification of frequency-modulated continuous wave (FMCW) LiDAR signals. This marks the inaugural use of the system for the identification of FMCW LiDAR signals. We elaborate on the fundamental principles of a delay-based optical RC system using an optical-injected distributed feedback laser (DFB) laser and discriminate four FMCW LiDAR signals through this setup. In the output layer, three distinct training algorithms—namely linear regression, support vector machine (SVM), and random forest—were employed to train the optical reservoir. Upon analyzing the experimental results, it was found that regardless of the size of the dataset, the recognition accuracy of the two nonlinear training algorithms was superior to that of the linear regression algorithm. Among the two nonlinear algorithms, the Random Forest algorithm had a higher recognition accuracy than SVM when the sample size was relatively small

Probabilistic Wind Power and Electricity Load Forecasting with Echo State Networks

With the introduction of distributed generation and the establishment of smart grids, several new challenges in energy analytics arose. These challenges can be solved with a specific type of recurrent neural networks called echo state networks, which can handle the combination of both weather and power consumption or production depending on the dataset to make predictions. Echo state networks are particularly suitable for time series forecasting tasks. Having accurate energy forecasts is paramount to assure grid operation and power provision remains reliable during peak hours when the consumption is high. The majority of load forecasting algorithms do not produce prediction intervals with coverage guarantees but rather produce simple point estimates. Information about uncer- tainty and prediction intervals is rarely useless. It helps grid operators change strategies for configuring the grid from conservative to risk-based ones and assess the reliability of operations. A popular way of producing prediction intervals in regression tasks is by applying Bayesian regression as the regression algorithm. As Bayesian regression is done by sampling, it nat- urally lends itself to generating intervals. However, Bayesian regression is not guaranteed to satisfy the designed coverage level for finite samples. This thesis aims to modify the traditional echo state network model to produce marginally valid and calibrated prediction intervals. This is done by replacing the standard linear regression method with Bayesian linear regression while simultaneously reducing the di- mensions to speed up the computation times. Afterward, a novel calibration technique for time series forecasting is applied in order to obtain said valid prediction intervals. The experiments are conducted using three different time series, two of them being a time series of electricity load. One is univariate, and the other is bivariate. The third time series is a wind power production time series. The proposed method showed promising results for all three datasets while significantly reducing computation times in the sampling ste

Learning regulatory compliance data for data governance in financial services industry by machine learning models

While regulatory compliance data has been governed in the financial services industry for a long time to identify, assess, remediate and prevent risks, improving data governance (“DG”) has emerged as a new paradigm that uses machine learning models to enhance the level of data management. In the literature, there is a research gap. Machine learning models have not been extensively applied to DG processes by a) predicting data quality (“DQ”) in supervised learning and taking temporal sequences and correlations of data noise into account in DQ prediction; b) predicting DQ in unsupervised learning and learning the importance of data noise jointly with temporal sequences and correlations of data noise in DQ prediction; c) analyzing DQ prediction at a granular level; d) measuring network run-time saving in DQ prediction; and e) predicting information security compliance levels. Our main research focus is whether our ML models accurately predict DQ and information security compliance levels during DG processes of financial institutions by learning regulatory compliance data from both theoretical and experimental perspectives. We propose five machine learning models including a) a DQ prediction sequential learning model in supervised learning; b) a DQ prediction sequential learning model with an attention mechanism in unsupervised learning; c) a DQ prediction analytical model; d) a DQ prediction network efficiency improvement model; and e) an information security compliance prediction model. Experimental results demonstrate the effectiveness of these models by accurately predicting DQ in supervised learning, precisely predicting DQ in unsupervised learning, analyzing DQ prediction by divergent dimensions such as risk types and business segments, saving significant network run-time in DQ prediction for improving the network efficiency, and accurately predicting information security compliance levels. Our models strengthen DG capabilities of financial institutions by improving DQ, data risk management, bank-wide risk management, and information security based on regulatory requirements in the financial services industry including Basel Committee on Banking Supervision Standard Number 239, Australia Prudential Regulation Authority (“APRA”) Standard Number CPG 235 and APRA Standard Number CPG 234. These models are part of DG programs under the DG framework of financial institutions