Training Echo State Networks with Regularization through Dimensionality Reduction

In this paper we introduce a new framework to train an Echo State Network to predict real valued time-series. The method consists in projecting the output of the internal layer of the network on a space with lower dimensionality, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework. Finally, we provide an insight on the effectiveness of the implemented mechanics through a visualization of the trajectory in the phase space and relying on the methodologies of nonlinear time-series analysis. By applying our method on well known chaotic systems, we provide evidence that the lower dimensional embedding retains the dynamical properties of the underlying system better than the full-dimensional internal states of the network

A Bayesian approach to simultaneously characterize the stochastic and deterministic components of a system

The present work provides a Bayesian approach to learn plausible models capable of characterizing complex time series in which deterministic and stochastic phenomena concur. Two main approaches are actually developed. The first approach, is a simple superposition model grounded on the hypothesis that the interactions between the stochastic and deterministic phenomena are negligible. To enable this model to capture complex dynamics, the stochastic part is assumed to be a fractal signal. Under the assumptions of this model, an analysis method is proposed, enabling the characterization of the fractal stochastic component and the estimation the deterministic part. The second main approach relies on Stochastic Differential Equations (SDEs) to model systems where the stochastic and deterministic part interact. First, a non-parametric estimation method for SDEs is developed, using recent advances from Gaussian processes. Finally, the thesis studies how to overcome the main constraint that the use of SDEs imposes: the Markovianity assumption. To that end, a new structured variational autoencoder with latent SDE dynamics is proposed. All the methods are tested on both synthetic and real signals, demonstrating its ability to capture the behavior of complex systems

Intrinsic Dimension Estimation: Relevant Techniques and a Benchmark Framework

When dealing with datasets comprising high-dimensional points, it is usually advantageous to discover some data structure. A fundamental information needed to this aim is the minimum number of parameters required to describe the data while minimizing the information loss. This number, usually called intrinsic dimension, can be interpreted as the dimension of the manifold from which the input data are supposed to be drawn. Due to its usefulness in many theoretical and practical problems, in the last decades the concept of intrinsic dimension has gained considerable attention in the scientific community, motivating the large number of intrinsic dimensionality estimators proposed in the literature. However, the problem is still open since most techniques cannot efficiently deal with datasets drawn from manifolds of high intrinsic dimension and nonlinearly embedded in higher dimensional spaces. This paper surveys some of the most interesting, widespread used, and advanced state-of-the-art methodologies. Unfortunately, since no benchmark database exists in this research field, an objective comparison among different techniques is not possible. Consequently, we suggest a benchmark framework and apply it to comparatively evaluate relevant state-of-the-art estimators

Analysis of Embodied and Situated Systems from an Antireductionist Perspective

The analysis of embodied and situated agents form a dynamical system perspective is often limited to a geometrical and qualitative description. However, a quantitative analysis is necessary to achieve a deep understanding of cognitive facts. The field of embodied cognition is multifaceted, and the first part of this thesis is devoted to exploring the diverse meanings proposed in the existing literature. This is a preliminary fundamental step as the creation of synthetic models requires well-founded theoretical and foundational boundaries for operationalising the concept of embodied and situated cognition in a concrete neuro-robotic model. By accepting the dynamical system view the agent is conceived as highly integrated and strictly coupled with the surrounding environment. Therefore the antireductionist framework is followed during the analysis of such systems, using chaos theory to unveil global properties and information theory to describe the complex network of interactions among the heterogeneous sub-components. In the experimental section, several evolutionary robotics experiments are discussed. This class of adaptive systems is consistent with the proposed definition of embodied and situated cognition. In fact, such neuro-robotics platforms autonomously develop a solution to a problem exploiting the continuous sensorimotor interaction with the environment. The first experiment is a stress test for chaos theory, a mathematical framework that studies erratic behaviour in low-dimensional and deterministic dynamical systems. The recorded dataset consists of the robots’ position in the environment during the execution of the task. Subsequently, the time series is projected onto a multidimensional phase space in order to study the underlying dynamic using chaotic numerical descriptors. Finally, such measures are correlated and confronted with the robots’ behavioural strategy and the performance in novel and unpredictable environments. The second experiment explores the possible applications of information-theoretic measures for the analysis of embodied and situated systems. Data is recorded from perceptual and motor neurons while robots are executing a wall-following task and pairwise estimations of the mutual information and the transfer entropy are calculated in order to create an exhaustive map of the nonlinear interactions among variables. Results show that the set of information-theoretic employed in this study unveils characteristics of the agent-environemnt interaction and the functional neural structure. This work aims at testing the explanatory power and impotence of nonlinear time series analysis applied to observables recorded from neuro-robotics embodied and situated systems

Variational Downscaling, Fusion and Assimilation of Hydrometeorological States via Regularized Estimation

Improved estimation of hydrometeorological states from down-sampled observations and background model forecasts in a noisy environment, has been a subject of growing research in the past decades. Here, we introduce a unified framework that ties together the problems of downscaling, data fusion and data assimilation as ill-posed inverse problems. This framework seeks solutions beyond the classic least squares estimation paradigms by imposing proper regularization, which are constraints consistent with the degree of smoothness and probabilistic structure of the underlying state. We review relevant regularization methods in derivative space and extend classic formulations of the aforementioned problems with particular emphasis on hydrologic and atmospheric applications. Informed by the statistical characteristics of the state variable of interest, the central results of the paper suggest that proper regularization can lead to a more accurate and stable recovery of the true state and hence more skillful forecasts. In particular, using the Tikhonov and Huber regularization in the derivative space, the promise of the proposed framework is demonstrated in static downscaling and fusion of synthetic multi-sensor precipitation data, while a data assimilation numerical experiment is presented using the heat equation in a variational setting