LTI ODE-valued neural networks adaptation of the back propagation algorithm

In [Velasco et al., 2014], a new approach of the classical artificial neural network archi-tecture is introduced, named ’LTI ODE-valued neural networks’, whereLTI ODEstandsfor Linear Time Invariant Ordinal Differential Equation. In this novel system, nodes inthe artificial neural network are characterized by: inputs in the form of differentiablecontinuous-time signals; linear time-invariant ordinary differential equations (LTI ODE)as connection weights; and activation functions evaluated in the frequency domain.It was shown that this new configuration allows solving multiple problems at the sametime using a common neural structure. However, the article concludes with the need fordeveloping learning algorithms for the new model of neural network.Taking as starting point the drawback pointed out in [Velasco et al., 2014], the mainobjective of this master thesis is to develop a training algorithm for a LTI ODE-valuedneural network. As a first and natural approach, modifications of the BackPropagationalgorithm is considered as a general framework. Moreover, since the nature of the inputsare differentiable continuous-time signals, it is analyzed how to obtain a model that canbe physically implemented in the form of an analogical circui

A brief network analysis of Artificial Intelligence publication

In this paper, we present an illustration to the history of Artificial Intelligence(AI) with a statistical analysis of publish since 1940. We collected and mined through the IEEE publish data base to analysis the geological and chronological variance of the activeness of research in AI. The connections between different institutes are showed. The result shows that the leading community of AI research are mainly in the USA, China, the Europe and Japan. The key institutes, authors and the research hotspots are revealed. It is found that the research institutes in the fields like Data Mining, Computer Vision, Pattern Recognition and some other fields of Machine Learning are quite consistent, implying a strong interaction between the community of each field. It is also showed that the research of Electronic Engineering and Industrial or Commercial applications are very active in California. Japan is also publishing a lot of papers in robotics. Due to the limitation of data source, the result might be overly influenced by the number of published articles, which is to our best improved by applying network keynode analysis on the research community instead of merely count the number of publish.Comment: 18 pages, 7 figure

An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysis

We investigate the relationship between system identification and intervention design in dynamical systems. While previous research demonstrated how identifiable representation learning methods, such as Independent Component Analysis (ICA), can reveal cause-effect relationships, it relied on a passive perspective without considering how to collect data. Our work shows that in Gaussian Linear Time-Invariant (LTI) systems, the system parameters can be identified by introducing diverse intervention signals in a multi-environment setting. By harnessing appropriate diversity assumptions motivated by the ICA literature, our findings connect experiment design and representational identifiability in dynamical systems. We corroborate our findings on synthetic and (simulated) physical data. Additionally, we show that Hidden Markov Models, in general, and (Gaussian) LTI systems, in particular, fulfil a generalization of the Causal de Finetti theorem with continuous parameters.Comment: CLeaR2024 camera ready. Code available at https://github.com/rpatrik96/lti-ic

NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.Comment: NIPS 201

Multiple Faults Estimation in Dynamical Systems: Tractable Design and Performance Bounds

In this article, we propose a tractable nonlinear fault isolation filter along with explicit performance bounds for a class of nonlinear dynamical systems. We consider the presence of additive and multiplicative faults, occurring simultaneously and through an identical dynamical relationship, which represents a relevant case in several application domains. The proposed filter architecture combines tools from model-based approaches in the control literature and regression techniques from machine learning. To this end, we view the regression operator through a system-theoretic perspective to develop operator bounds that are then utilized to derive performance bounds for the proposed estimation filter. In the case of constant, simultaneously and identically acting additive and multiplicative faults, it can be shown that the estimation error converges to zero with an exponential rate. The performance of the proposed estimation filter in the presence of incipient faults is validated through an application on the lateral safety systems of SAE level 4 automated vehicles. The numerical results show that the theoretical bounds of this study are indeed close to the actual estimation error.Comment: 24 pages, 8 figure

Robust identification of non-autonomous dynamical systems using stochastic dynamics models

This paper considers the problem of system identification (ID) of linear and nonlinear non-autonomous systems from noisy and sparse data. We propose and analyze an objective function derived from a Bayesian formulation for learning a hidden Markov model with stochastic dynamics. We then analyze this objective function in the context of several state-of-the-art approaches for both linear and nonlinear system ID. In the former, we analyze least squares approaches for Markov parameter estimation, and in the latter, we analyze the multiple shooting approach. We demonstrate the limitations of the optimization problems posed by these existing methods by showing that they can be seen as special cases of the proposed optimization objective under certain simplifying assumptions: conditional independence of data and zero model error. Furthermore, we observe that our proposed approach has improved smoothness and inherent regularization that make it well-suited for system ID and provide mathematical explanations for these characteristics' origins. Finally, numerical simulations demonstrate a mean squared error over 8.7 times lower compared to multiple shooting when data are noisy and/or sparse. Moreover, the proposed approach can identify accurate and generalizable models even when there are more parameters than data or when the underlying system exhibits chaotic behavior