Neural network for estimating and compensating the nonlinear characteristics of nonstationary complex systems

Issued as final reportNational Science Foundation (U.S

A Comparative Study of Reservoir Computing for Temporal Signal Processing

Reservoir computing (RC) is a novel approach to time series prediction using recurrent neural networks. In RC, an input signal perturbs the intrinsic dynamics of a medium called a reservoir. A readout layer is then trained to reconstruct a target output from the reservoir's state. The multitude of RC architectures and evaluation metrics poses a challenge to both practitioners and theorists who study the task-solving performance and computational power of RC. In addition, in contrast to traditional computation models, the reservoir is a dynamical system in which computation and memory are inseparable, and therefore hard to analyze. Here, we compare echo state networks (ESN), a popular RC architecture, with tapped-delay lines (DL) and nonlinear autoregressive exogenous (NARX) networks, which we use to model systems with limited computation and limited memory respectively. We compare the performance of the three systems while computing three common benchmark time series: H{\'e}non Map, NARMA10, and NARMA20. We find that the role of the reservoir in the reservoir computing paradigm goes beyond providing a memory of the past inputs. The DL and the NARX network have higher memorization capability, but fall short of the generalization power of the ESN

Modeling neural plasticity in echo state networks for time series prediction

In this paper, we investigate the influence of neural plasticity on the learning performance of echo state networks (ESNs) and supervised learning algorithms in training readout connections for two time series prediction problems including the sunspot time series and the Mackey Glass chaotic system. We implement two different plasticity rules that are expected to improve the prediction performance, namely, anti-Oja learning rule and the Bienenstock-Cooper-Munro (BCM) learning rule combined with both offline and online learning of the readout connections. Our experimental results have demonstrated that the neural plasticity can more significantly enhance the learning in offline learning than in online learning

Digital Twins and Artificial Intelligence for Applications in Electric Power Distribution Systems

As modern electric power distribution systems (MEPDS) continue to grow in complexity, largely due to the ever-increasing penetration of Distributed Energy Resources (DERs), particularly solar photovoltaics (PVs) at the distribution level, there is a need to facilitate advanced operational and management tasks in the system driven by this complexity, especially in systems with high renewable penetration dependent on complex weather phenomena. Digital twins (DTs), or virtual replicas of the system and its assets, enhanced with AI paradigms can add enormous value to tasks performed by regulators, distribution system operators and energy market analysts, thereby providing cognition to the system. DTs of MEPDS assets and the system can be utilized for real-time and faster-than-real-time operational and management task support, planning studies, scenario analysis, data analytics and other distribution system studies. This study leverages DT and AI to enhance DER integration in an MEPDS as well as operational and management (O&M) tasks and distribution system studies based on a system with high PV penetration. DTs have been used to both estimate and predict the behavior of an existing 1 MW plant in Clemson University by developing asset digital twins of the physical system. Solar irradiance, temperature and wind-speed variations in the area have been modeled using physical weather stations located in and around the Clemson region to develop ten virtual weather stations. Finally, DTs of the system along with virtual and physical weather stations are used to both estimate and predict, in short time intervals, the real-time behavior of potential PV plant installations over the region. Ten virtual PV plants and three hybrid PV plants are studied, for enhanced cognition in the system. These physical, hybrid and virtual PV sources enable situational awareness and situational intelligence of real-time PV production in a distribution system

Echo state network‐based feature extraction for efficient color image segmentation

Image segmentation plays a crucial role in many image processing and understanding applications. Despite the huge number of proposed image segmentation techniques, accurate segmentation remains a significant challenge in image analysis. This article investigates the viability of using echo state network (ESN), a biologically inspired recurrent neural network, as features extractor for efficient color image segmentation. First, an ensemble of initial pixel features is extracted from the original images and injected into the ESN reservoir. Second, the internal activations of the reservoir neurons are used as new pixel features. Third, the new features are classified using a feed forward neural network as a readout layer for the ESN. The quality of the pixel features produced by the ESN is evaluated through extensive series of experiments conducted on real world image datasets. The optimal operating range of different ESN setup parameters for producing competitive quality features is identified. The performance of the proposed ESN‐based framework is also evaluated on a domain‐specific application, namely, blood vessel segmentation in retinal images where experiments are conducted on the widely used digital retinal images for vessel extraction (DRIVE) dataset. The obtained results demonstrate that the proposed method outperforms state‐of‐the‐art general segmentation techniques in terms of performance with an F‐score of 0.92 ± 0.003 on the segmentation evaluation dataset. In addition, the proposed method achieves a comparable segmentation accuracy (0.9470) comparing with reported techniques of segmentation of blood vessels in images of retina and outperform them in terms of processing time. The average time required by our technique to segment one retinal image from DRIVE dataset is 8 seconds. Furthermore, empirically derived guidelines are proposed for adequately setting the ESN parameters for effective color image segmentation

Sistema preditivo baseado em redes recorrentes para controle de reatores nucleares de pesquisa

The control of nuclear reactors is done by systems that monitor several variables simultaneously, but only indicate the occurrence of an accident, without identifying its type or cause. The possibility of prediction of an accident based on the evolution of the current variables of the reactor can be a very useful tool to assist the work of the reactor operators, increasing their confidence, as well as the general availability of the plant. Echo State Networks (ESN) are Recurrent Neural Networks (RNNs) suitable for the formulation of predictive systems, with the advantage of using a training algorithm that is simpler and faster than conventional RNNs. To use Echo State Networks in predictive systems it is not necessary to specify the structure of the model, but some parameters of this network must be tuned for good performance. This work proposes the use of Artificial Intelligence to optimize ESN’s parameters, to improve performance. Particle Swarm Optimization (PSO), a method that uses so-called swarm intelligence, was used. The ESN optimized by the PSO was used to monitor the operation of a nuclear reactor (with real operating data), as well as to identify in real time if an accident is occurring, as well which accident is occurring. The results show that the network can identify and predict the state of the reactor, be it in normal or accident condition.O controle de reatores nucleares é feito por sistemas que monitoram diversas variáveis simultaneamente, porém apenas apontam a ocorrência de um acidente, sem identificar seu tipo ou causa. O poder de previsão de que um acidente possa ocorrer baseado na evolução das variáveis atuais do reator pode ser uma ferramenta bastante útil para auxiliar o trabalho dos operadores do reator, aumentando não só sua confiança, mas também a disponibilidade geral da planta. As redes Echo State Network (ESN) são Redes Neurais Recorrentes (RNR) apropriadas para a formulação de sistemas preditivos, com a vantagem de utilizarem um algoritmo de treinamento muito mais simples e mais rápido que as RNR convencionais. Porém, para utilização das ESN em sistemas preditivos não é necessário especificar a estrutura do modelo, porém alguns parâmetros desta rede devem ser trabalhados e ajustados para se obter um bom desempenho. Este trabalho propõe a utilização de Inteligência Artificial para otimização dos parâmetros da ESN, a fim de se obter o melhor desempenho possível. Foi utilizada a Otimização por Enxame de Partículas (PSO - Particle Swarm Optimization), método que utiliza a chamada inteligência de enxame, que evolui a população em busca dos melhores resultados. A rede ESN com seu treinamento otimizado pelo PSO foi utilizada para acompanhar o funcionamento de um reator nuclear, com dados reais de funcionamento, assim como identificar em tempo real se algum acidente está ocorrendo, bem como qual acidente está ocorrendo. Os resultados obtidos demonstram que a rede pode identificar e prever, o estado do reator, seja este em condição normal ou de acidente

On-device mobile speech recognition

Despite many years of research, Speech Recognition remains an active area of research in Artificial Intelligence. Currently, the most common commercial application of this technology on mobile devices uses a wireless client – server approach to meet the computational and memory demands of the speech recognition process. Unfortunately, such an approach is unlikely to remain viable when fully applied over the approximately 7.22 Billion mobile phones currently in circulation. In this thesis we present an On – Device Speech recognition system. Such a system has the potential to completely eliminate the wireless client-server bottleneck. For the Voice Activity Detection part of this work, this thesis presents two novel algorithms used to detect speech activity within an audio signal. The first algorithm is based on the Log Linear Predictive Cepstral Coefficients Residual signal. These LLPCCRS feature vectors were then classified into voice signal and non-voice signal segments using a modified K-means clustering algorithm. This VAD algorithm is shown to provide a better performance as compared to a conventional energy frame analysis based approach. The second algorithm developed is based on the Linear Predictive Cepstral Coefficients. This algorithm uses the frames within the speech signal with the minimum and maximum standard deviation, as candidates for a linear cross correlation against the rest of the frames within the audio signal. The cross correlated frames are then classified using the same modified K-means clustering algorithm. The resulting output provides a cluster for Speech frames and another cluster for Non–speech frames. This novel application of the linear cross correlation technique to linear predictive cepstral coefficients feature vectors provides a fast computation method for use on the mobile platform; as shown by the results presented in this thesis. The Speech recognition part of this thesis presents two novel Neural Network approaches to mobile Speech recognition. Firstly, a recurrent neural networks architecture is developed to accommodate the output of the VAD stage. Specifically, an Echo State Network (ESN) is used for phoneme level recognition. The drawbacks and advantages of this method are explained further within the thesis. Secondly, a dynamic Multi-Layer Perceptron approach is developed. This builds on the drawbacks of the ESN and provides a dynamic way of handling speech signal length variabilities within its architecture. This novel Dynamic Multi-Layer Perceptron uses both the Linear Predictive Cepstral Coefficients (LPC) and the Mel Frequency Cepstral Coefficients (MFCC) as input features. A speaker dependent approach is presented using the Centre for spoken Language and Understanding (CSLU) database. The results show a very distinct behaviour from conventional speech recognition approaches because the LPC shows performance figures very close to the MFCC. A speaker independent system, using the standard TIMIT dataset, is then implemented on the dynamic MLP for further confirmation of this. In this mode of operation the MFCC outperforms the LPC. Finally, all the results, with emphasis on the computation time of both these novel neural network approaches are compared directly to a conventional hidden Markov model on the CSLU and TIMIT standard datasets

On the induction of temporal structure by recurrent neural networks

Language acquisition is one of the core problems in artificial intelligence (AI) and it is generally accepted that any successful AI account of the mind will stand or fall depending on its ability to model human language. Simple Recurrent Networks (SRNs) are a class of so-called artificial neural networks that have a long history in language modelling via learning to predict the next word in a sentence. However, SRNs have also been shown to suffer from catastrophic forgetting, lack of syntactic systematicity and an inability to represent more than three levels of centre-embedding, due to the so-called 'vanishing gradients' problem. This problem is caused by the decay of past input information encoded within the error-gradients which vanish exponentially as additional input information is encountered and passed through the recurrent connections. That said, a number of architectural variations have been applied which may compensate for this issue, such as the Nonlinear Autoregressive Network with exogenous inputs (NARX) network and the multi-recurrent network (MRN). In addition to this, Echo State Networks (ESNs) are a relatively new class of recurrent neural network that do not suffer from the vanishing gradients problem and have been shown to exhibit state-of-the-art performance in tasks such as motor control, dynamic time series prediction, and more recently language processing. This research re-explores the class of SRNs and evaluates them against the state-of-the-art ESN to identify which model class is best able to induce the underlying finite-state automaton of the target grammar implicitly through the next word prediction task. In order to meet its aim, the research analyses the internal representations formed by each of the different models and explores the conditions under which they are able to carry information about long term sequential dependencies beyond what is found in the training data. The findings of the research are significant. It reveals that the traditional class of SRNs, trained with backpropagation through time, are superior to ESNs for the grammar prediction task. More specifically, the MRN, with its state-based memory of varying rigidity, is more able to learn the underlying grammar than any other model. An analysis of the MRN’s internal state reveals that this is due to its ability to maintain a constant variance within its state-based representation of the embedded aspects (or finite state machines) of the target grammar. The investigations show that in order to successfully induce complex context free grammars directly from sentence examples, then not only are a hidden layer and output layer recurrency required, but so is self-recurrency on the context layer to enable varying degrees of current and past state information, that are integrated over time

Reservoir Computing for Learning in Structured Domains

The study of learning models for direct processing complex data structures has gained an increasing interest within the Machine Learning (ML) community during the last decades. In this concern, efficiency, effectiveness and adaptivity of the ML models on large classes of data structures represent challenging and open research issues. The paradigm under consideration is Reservoir Computing (RC), a novel and extremely efficient methodology for modeling Recurrent Neural Networks (RNN) for adaptive sequence processing. RC comprises a number of different neural models, among which the Echo State Network (ESN) probably represents the most popular, used and studied one. Another research area of interest is represented by Recursive Neural Networks (RecNNs), constituting a class of neural network models recently proposed for dealing with hierarchical data structures directly. In this thesis the RC paradigm is investigated and suitably generalized in order to approach the problems arising from learning in structured domains. The research studies described in this thesis cover classes of data structures characterized by increasing complexity, from sequences, to trees and graphs structures. Accordingly, the research focus goes progressively from the analysis of standard ESNs for sequence processing, to the development of new models for trees and graphs structured domains. The analysis of ESNs for sequence processing addresses the interesting problem of identifying and characterizing the relevant factors which influence the reservoir dynamics and the ESN performance. Promising applications of ESNs in the emerging field of Ambient Assisted Living are also presented and discussed. Moving towards highly structured data representations, the ESN model is extended to deal with complex structures directly, resulting in the proposed TreeESN, which is suitable for domains comprising hierarchical structures, and Graph-ESN, which generalizes the approach to a large class of cyclic/acyclic directed/undirected labeled graphs. TreeESNs and GraphESNs represent both novel RC models for structured data and extremely efficient approaches for modeling RecNNs, eventually contributing to the definition of an RC framework for learning in structured domains. The problem of adaptively exploiting the state space in GraphESNs is also investigated, with specific regard to tasks in which input graphs are required to be mapped into flat vectorial outputs, resulting in the GraphESN-wnn and GraphESN-NG models. As a further point, the generalization performance of the proposed models is evaluated considering both artificial and complex real-world tasks from different application domains, including Chemistry, Toxicology and Document Processing