Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this record.In this paper, we elaborate over the well-known interpretability issue in echo-state networks (ESNs). The idea is to investigate the dynamics of reservoir neurons with time-series analysis techniques developed in complex systems research. Notably, we analyze time series of neuron activations with recurrence plots (RPs) and recurrence quantification analysis (RQA), which permit to visualize and characterize high-dimensional dynamical systems. We show that this approach is useful in a number of ways. First, the 2-D representation offered by RPs provides a visualization of the high-dimensional reservoir dynamics. Our results suggest that, if the network is stable, reservoir and input generate similar line patterns in the respective RPs. Conversely, as the ESN becomes unstable, the patterns in the RP of the reservoir change. As a second result, we show that an RQA measure, called Lmax, is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution can quantify network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We leverage on this property to determine the edge of stability and show that our criterion is more accurate than two well-known counterparts, both based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses are valuable tools to design an ESN, given a specific problem

Parameterizing and Aggregating Activation Functions in Deep Neural Networks

The nonlinear activation functions applied by each neuron in a neural network are essential for making neural networks powerful representational models. If these are omitted, even deep neural networks reduce to simple linear regression due to the fact that a linear combination of linear combinations is still a linear combination. In much of the existing literature on neural networks, just one or two activation functions are selected for the entire network, even though the use of heterogenous activation functions has been shown to produce superior results in some cases. Even less often employed are activation functions that can adapt their nonlinearities as network parameters along with standard weights and biases. This dissertation presents a collection of papers that advance the state of heterogenous and parameterized activation functions. Contributions of this dissertation include three novel parametric activation functions and applications of each, a study evaluating the utility of the parameters in parametric activation functions, an aggregated activation approach to modeling time-series data as an alternative to recurrent neural networks, and an improvement upon existing work that aggregates neuron inputs using product instead of sum

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

In this paper, we elaborate over the well-known interpretability issue in echo-state networks (ESNs). The idea is to investigate the dynamics of reservoir neurons with time-series analysis techniques developed in complex systems research. Notably, we analyze time series of neuron activations with recurrence plots (RPs) and recurrence quantification analysis (RQA), which permit to visualize and characterize high-dimensional dynamical systems. We show that this approach is useful in a number of ways. First, the 2-D representation offered by RPs provides a visualization of the high-dimensional reservoir dynamics. Our results suggest that, if the network is stable, reservoir and input generate similar line patterns in the respective RPs. Conversely, as the ESN becomes unstable, the patterns in the RP of the reservoir change. As a second result, we show that an RQA measure, called Lmax, is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution can quantify network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We leverage on this property to determine the edge of stability and show that our criterion is more accurate than two well-known counterparts, both based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses are valuable tools to design an ESN, given a specific problem

ESTIMATION AND CONTROL OF NONLINEAR SYSTEMS: MODEL-BASED AND MODEL-FREE APPROACHES

State estimation and subsequent controller design for a general nonlinear system is an important problem that have been studied over the past decades. Many applications, e.g., atmospheric and oceanic sampling or lift control of an airfoil, display strongly nonlinear dynamics with very high dimensionality. Some of these applications use smaller underwater or aerial sensing platforms with insufficient on-board computation power to use a Monte-Carlo approach of particle filters. Hence, they need a computationally efficient filtering method for state-estimation without a severe penalty on the performance. On the other hand, the difficulty of obtaining a reliable model of the underlying system, e.g., a high-dimensional fluid dynamical environment or vehicle flow in a complex traffic network, calls for the design of a data-driven estimation and controller when abundant measurements are present from a variety of sensors. This dissertation places these problems in two broad categories: model-based and model-free estimation and output feedback. In the first part of the dissertation, a semi-parametric method with Gaussian mixture model (GMM) is used to approximate the unknown density of states. Then a Kalman filter and its nonlinear variants are employed to propagate and update each Gaussian mode with a Bayesian update rule. The linear observation model permits a Kalman filter covariance update for each Gaussian mode. The estimation error is shown to be stochastically bounded and this is illustrated numerically. The estimate is used in an observer-based feedback control to stabilize a general closed-loop system. A transferoperator- based approach is then proposed for the motion update for Bayesian filtering of a nonlinear system. A finite-dimensional approximation of the Perron-Frobenius (PF) operator yields a method called constrained Ulam dynamic mode decomposition (CUDMD). This algorithm is applied for output feedback of a pitching airfoil in unsteady flow. For the second part, an echo-state network (ESN) based approach equipped with an ensemble Kalman filter is proposed for data-driven estimation of a nonlinear system from a time series. A random reservoir of recurrent neural connections with the echo-state property (ESP) is trained from a time-series data. It is then used as a model-predictor for an ensemble Kalman filter for sparse estimation. The proposed data-driven estimation method is applied to predict the traffic flow from a set of mobility data of the UMD campus. A data-driven model-identification and controller design is also developed for control-affine nonlinear systems that are ubiquitous in several aerospace applications. We seek to find an approximate linear/bilinear representation of these nonlinear systems from data using the extended dynamic mode decomposition algorithm (EDMD) and apply Liealgebraic methods to analyze the controllability and design a controller. The proposed method utilizes the Koopman canonical transform (KCT) to approximate the dynamics into a bilinear system (Koopman bilinear form) under certain assumptions. The accuracy of this approximation is then analytically justified with the universal approximation property of the Koopman eigenfunctions. The resulting bilinear system is then subjected to controllability analysis using the Myhill semigroup and Lie algebraic structures, and a fixed endpoint optimal controller is designed using the Pontryagin’s principle

Approximately periodic time series and nonlinear structures

In this thesis a previously developed framework for modelling diversity of approximately periodic time series is considered. In this framework the diversity is modelled deterministically, exploiting the irregularity of chaos. This is an alternative to other well established frameworks which use probability distributions and other stochastic tools to describe diversity. The diversity which is to be modelled, on the other hand, is not assumed to be of chaotic nature, but can stem out from a stochastic process, though it has never been verified before, whether or not purely stochastic patterns can be modelled that way. The main application of such a modelling technique would be pattern recognition; once a model for a learning set of approximately periodic time series is found, synchronisation-like phenomena could be used to determine if a novel time series is similar to the members of the learning set. The most crucial step of the classification procedure outlined before is the automatic generation of a chaotic model from data, called identification. Originally, this was done using a simple low dimensional reference model. Here, on the other hand, a biologically inspired approach is taken. This has the advantage that the identification and classification procedure could be greatly simplified and the computational power involved significantly reduced. The biologically inspired model used for identification was announced several time in literature under the name "Echo State Network". The articles available on it consisted mainly of examples were it performed remarkably well, though a thorough analysis was still missing to the scientific community. Here the model is analysed using a measure that had appeared already in similar contexts and with help of this measure good settings of the models' parameter were determined. Finally, the model was used to assess if stochastic patterns can be modelled by chaotic signals. Indeed, it has been shown that, for the biologically inspired modelling technique considered, chaotic behaviour appears to implicitly model diversity and randomness of the learnt patterns whenever these are sufficiently structured; whilst chaos does not appear when the patterns are remarkably unstructured. In other words, deterministic chaos or strongly coloured noise lead to the chaos emergence as opposed to white-like noise which does not. With this result in mind, the classification of gait signals was attempted, as no signs of chaoticity could be found in them and the previously available modelling technique seemed to have difficulties to model their diversity. The identification and classification results with the biologically inspired model turned out to be very good

Dynamic element matching techniques for data converters

Analog to digital converter (ADC) circuit component errors create nonuniform quantization code widths and create harmonic distortion in an ADC\u27s output. In this dissertation, two techniques for estimating an ADC\u27s output spectrum from the ADC\u27s transfer function are determined. These methods are compared to a symmetric power function and asymmetric power function approximations. Standard ADC performance metrics, such as SDR, SNDR, SNR, and SFDR, are also determined as a function of the ADC\u27s transfer function approximations. New dynamic element matching (DEM) flash ADCs are developed. An analysis of these DEM flash ADCs is developed and shows that these DEM algorithms improve an ADC\u27s performance. The analysis is also used to analyze several existing DEM ADC architectures; Digital to analog converter (DAC) circuit component errors create nonuniform quantization code widths and create harmonic distortion in a DAC\u27s output. In this dissertation, an exact relationship between a DAC\u27s integral nonlinearity (INL) and its output spectrum is determined. Using this relationship, standard DAC performance metrics, such as SDR, SNDR, SNR, and SFDR, are calculated from the DAC\u27s transfer function. Furthermore, an iterative method is developed which determines an arbitrary DAC\u27s transfer function from observed output magnitude spectra. An analysis of DEM techniques for DACs, including the determination of several suitable metrics by which DEM techniques can be compared, is derived. The performance of a given DEM technique is related to standard DAC performance metrics, such as SDR, SNDR, and SFDR. Conditions under which DEM techniques can guarantee zero average INL and render the distortion due to mismatched components as white noise are developed. Several DEM circuits proposed in the literature are shown to be equivalent and have hardware efficient implementations based on multistage interconnection networks. Example DEM circuit topologies and their hardware efficient VLSI implementations are also presented