Automatic Extraction of Ordinary Differential Equations from Data: Sparse Regression Tools for System Identification

Studying nonlinear systems across engineering, physics, economics, biology, and chemistry often hinges upon successfully discovering their underlying dynamics. However, despite the abundance of data in today's world, a complete comprehension of these governing equations often remains elusive, posing a significant challenge. Traditional system identification methods for building mathematical models to describe these dynamics can be time-consuming, error-prone, and limited by data availability. This thesis presents three comprehensive strategies to address these challenges and automate model discovery. The procedures outlined here employ classic statistical and machine learning methods, such as signal filtering, sparse regression, bootstrap sampling, Bayesian inference, and unsupervised learning algorithms, to capture complex and nonlinear relationships in data. Building on these foundational techniques, the proposed processes offer a reliable and efficient approach to identifying models of ordinary differential equations from data, differing from and complementing existing frameworks. The results presented here provide rigorous benchmarking against state-of-the-art algorithms, demonstrating the proposed methods' effectiveness in model discovery and highlighting the potential for discovering governing equations across applications such as weather forecasting, chemical reaction and electrical circuit modelling, and predator-prey dynamics. These methods can aid in solving critical decision-making problems, including optimising resource allocation, predicting system failures, and facilitating adaptive control in various domains. Ultimately, the strategies developed in this thesis are designed to integrate seamlessly into current workflows, thereby promoting data-driven decision-making and enhancing understanding of complex system dynamics

Moving horizon optimization methods, applications and tools for learning and controlling dynamical systems

Mathematical models based on dynamical systems are crucial for understanding complex phenomena across a wide range of scientific and engineering disciplines. Optimizing these models can significantly improve the performance (e.g., in the sense of socioeconomic, environmental, and safety concerns) of various processes and systems that support our modern society, such as e.g. supply chain networks and chemical manufacturing processes. However, controlling these systems in the presence of uncertainty and for high-dimensional models is challenging. Developing robust and efficient optimization models and solution algorithms for this purpose is therefore crucial. Similarly, optimization techniques can be used to infer the governing equations for such dynamical systems from available measurement data. Learning such models is important not only for performing the aforementioned control tasks, but also for advancing our understanding of the physical laws that govern the phenomena we have so long observed but cannot quantitatively explain. Motivated by the above, this dissertation contributes novel moving horizon optimization methods, applications and tools for learning and controlling a variety of dynamical systems. The first part of this dissertation introduces the background and theory of moving horizon estimation and control methods. As a motivating example, I present a novel application of these existing methods to the optimal data-driven management of the COVID-19 pandemic in the US. The proposed approach identifies optimal social distancing and testing policies that minimize socioeconomic impact, while keeping the the number of infected individuals under a specified threshold. Subsequently, I focus on dynamical system models corresponding networks of integrators for optimal supply chain management under uncertainty. The first methodological contribution corresponds to a tube-based robust economic model predictive control framework for sparse storage systems, which I shown to have improved feasibility for supply chain management under demand disturbances. The proposed approach significantly improves computational performance relative to the available methods. Subsequently, I develop an extensive and systematic case study evaluating the performance of deterministic (feedback-based, closed-loop, or online) moving horizon optimization in comparison to stochastic and robust methods for supply chain management under increasing levels of uncertainty, forecasting errors, and recourse availability. Having demonstrated the overall robust and computationally efficient performance of deterministic moving horizon optimization techniques, the second part of the dissertation is focused on a class of multi-scale dynamical systems corresponding to supply chains of highly perishable inventory. This type of supply chains require integration of the inventory management problem with quality control by manipulating environmental conditions (e.g., temperature) during shipment and storage, which directly impact the product deterioration rate. To this end, I introduce a novel modeling approach for incorporating complex, multivariate physico-chemical product quality dynamics within the supply chain inventory balances, and provide a computationally efficient reformulation thereof. Based on this modeling approach and the results introduced in Part I of the dissertation, I develop a stabilizing closed-loop optimal supply chain production and distribution planning framework to handle uncertainties, such as random customer demand and/or random product quality spoilage. I then propose a scalable solution heuristic approach to cope with larger supply chain networks, and I present several case studies to demonstrate robustness to demand uncertainty. Lastly, I develop a simultaneous state estimation and closed-loop control approach to account for the fact that product quality may not be completely measurable in practical settings. In the third and final part of the dissertation, the focus shifts from controlling dynamical systems to learning their governing equations from data via moving horizon optimization. Here, I develop methods based on dynamic nonlinear optimization which, compared to existing efforts, demonstrate greater flexibility for handling highly nonlinear systems, for incorporating prior domain knowledge, and coping with high amounts of measurement noise in the training data. I then demonstrate the extension of this learning framework to the case of reactive dynamical system and present numerical experiments for non-isothermal continuous and batch chemical reactors. Lastly, I develop a sequential dynamic nonlinear optimization approach for discovering and performing dimensionality reduction of microkinetic reaction networks.Chemical Engineerin

On data-selective learning

Adaptive filters are applied in several electronic and communication devices like smartphones, advanced headphones, DSP chips, smart antenna, and teleconference systems. Also, they have application in many areas such as system identification, channel equalization, noise reduction, echo cancellation, interference cancellation, signal prediction, and stock market. Therefore, reducing the energy consumption of the adaptive filtering algorithms has great importance, particularly in green technologies and in devices using battery. In this thesis, data-selective adaptive filters, in particular the set-membership (SM) adaptive filters, are the tools to reach the goal. There are well known SM adaptive filters in literature. This work introduces new algorithms based on the classical ones in order to improve their performances and reduce the number of required arithmetic operations at the same time. Therefore, firstly, we analyze the robustness of the classical SM adaptive filtering algorithms. Secondly, we extend the SM technique to trinion and quaternion systems. Thirdly, by combining SM filtering and partialupdating, we introduce a new improved set-membership affine projection algorithm with constrained step size to improve its stability behavior. Fourthly, we propose some new least-mean-square (LMS) based and recursive least-squares based adaptive filtering algorithms with low computational complexity for sparse systems. Finally, we derive some feature LMS algorithms to exploit the hidden sparsity in the parameters.Filtros adaptativos são aplicados em diversos aparelhos eletrônicos e de comunicação, como smartphones, fone de ouvido avançados, DSP chips, antenas inteligentes e sistemas de teleconferência. Eles também têm aplicação em várias áreas como identificação de sistemas, equalização de canal, cancelamento de eco, cancelamento de interferência, previsão de sinal e mercado de ações. Desse modo, reduzir o consumo de energia de algoritmos adaptativos tem importância significativa, especialmente em tecnologias verdes e aparelhos que usam bateria. Nesta tese, filtros adaptativos com seleção de dados, em particular filtros adaptativos da família set-membership (SM), são apresentados para cumprir essa missão. No presente trabalho objetivamos apresentar novos algoritmos, baseados nos clássicos, a fim de aperfeiçoar seus desempenhos e, ao mesmo tempo, reduzir o número de operações aritméticas exigidas. Dessa forma, primeiro analisamos a robustez dos filtros adaptativos SM clássicos. Segundo, estendemos o SM aos números trinions e quaternions. Terceiro, foram utilizadas também duas famílias de algoritmos, SM filtering e partial-updating, de uma maneira elegante, visando reduzir energia ao máximo possível e obter um desempenho competitivo em termos de estabilidade. Quarto, a tese propõe novos filtros adaptativos baseado em algoritmos least-mean-square (LMS) e mínimos quadrados recursivos com complexidade computacional baixa para espaços esparsos. Finalmente, derivamos alguns algoritmos feature LMS para explorar a esparsidade escondida nos parâmetros

International Conference on Continuous Optimization (ICCOPT) 2019 Conference Book

The Sixth International Conference on Continuous Optimization took place on the campus of the Technical University of Berlin, August 3-8, 2019. The ICCOPT is a flagship conference of the Mathematical Optimization Society (MOS), organized every three years. ICCOPT 2019 was hosted by the Weierstrass Institute for Applied Analysis and Stochastics (WIAS) Berlin. It included a Summer School and a Conference with a series of plenary and semi-plenary talks, organized and contributed sessions, and poster sessions. This book comprises the full conference program. It contains, in particular, the scientific program in survey style as well as with all details, and information on the social program, the venue, special meetings, and more

Bayesian Modeling and Estimation Techniques for the Analysis of Neuroimaging Data

Brain function is hallmarked by its adaptivity and robustness, arising from underlying neural activity that admits well-structured representations in the temporal, spatial, or spectral domains. While neuroimaging techniques such as Electroencephalography (EEG) and magnetoencephalography (MEG) can record rapid neural dynamics at high temporal resolutions, they face several signal processing challenges that hinder their full utilization in capturing these characteristics of neural activity. The objective of this dissertation is to devise statistical modeling and estimation methodologies that account for the dynamic and structured representations of neural activity and to demonstrate their utility in application to experimentally-recorded data. The first part of this dissertation concerns spectral analysis of neural data. In order to capture the non-stationarities involved in neural oscillations, we integrate multitaper spectral analysis and state-space modeling in a Bayesian estimation setting. We also present a multitaper spectral analysis method tailored for spike trains that captures the non-linearities involved in neuronal spiking. We apply our proposed algorithms to both EEG and spike recordings, which reveal significant gains in spectral resolution and noise reduction. In the second part, we investigate cortical encoding of speech as manifested in MEG responses. These responses are often modeled via a linear filter, referred to as the temporal response function (TRF). While the TRFs estimated from the sensor-level MEG data have been widely studied, their cortical origins are not fully understood. We define the new notion of Neuro-Current Response Functions (NCRFs) for simultaneously determining the TRFs and their cortical distribution. We develop an efficient algorithm for NCRF estimation and apply it to MEG data, which provides new insights into the cortical dynamics underlying speech processing. Finally, in the third part, we consider the inference of Granger causal (GC) influences in high-dimensional time series models with sparse coupling. We consider a canonical sparse bivariate autoregressive model and define a new statistic for inferring GC influences, which we refer to as the LASSO-based Granger Causal (LGC) statistic. We establish non-asymptotic guarantees for robust identification of GC influences via the LGC statistic. Applications to simulated and real data demonstrate the utility of the LGC statistic in robust GC identification