Model-based Reinforcement Learning of Nonlinear Dynamical Systems

Model-based Reinforcement Learning (MBRL) techniques accelerate the learning task by employing a transition model to make predictions. In this dissertation, we present novel techniques for online learning of unknown dynamics by iteratively computing a feedback controller based on the most recent update of the model. Assuming a structured continuous-time model of the system in terms of a set of bases, we formulate an infinite horizon optimal control problem addressing a given control objective. The structure of the system along with a value function parameterized in the quadratic form provides flexibility in analytically calculating an update rule for the parameters. Hence, a matrix differential equation of the parameters is obtained, where the solution is used to characterize the optimal feedback control in terms of the bases, at any time step. Moreover, the quadratic form of the value function suggests a compact way of updating the parameters that considerably decreases the computational complexity. In the convergence analysis, we demonstrate asymptotic stability and optimality of the obtained learning algorithm around the equilibrium by revealing its connections with the analogous Linear Quadratic Regulator (LQR). Moreover, the results are extended to the trajectory tracking problem. Assuming a structured unknown nonlinear system augmented with the dynamics of a commander system, we obtain a control rule minimizing a given quadratic tracking objective function. Furthermore, in an alternative technique for learning, a piecewise nonlinear affine framework is developed for controlling nonlinear systems with unknown dynamics. Therefore, we extend the results to obtain a general piecewise nonlinear framework where each piece is responsible for locally learning and controlling over some partition of the domain. Then, we consider the Piecewise Affine (PWA) system with a bounded uncertainty as a special case, for which we suggest an optimization-based verification technique. Accordingly, given a discretization of the learned PWA system, we iteratively search for a common piecewise Lyapunov function in a set of positive definite functions, where a non-monotonic convergence is allowed. Then, this Lyapunov candidate is verified for the uncertain system. To demonstrate the applicability of the approaches presented in this dissertation, simulation results on benchmark nonlinear systems are included, such as quadrotor, vehicle, etc. Moreover, as another detailed application, we investigate the Maximum Power Point Tracking (MPPT) problem of solar Photovoltaic (PV) systems. Therefore, we develop an analytical nonlinear optimal control approach that assumes a known model. Then, we apply the obtained nonlinear optimal controller together with the piecewise MBRL technique presented previously

An Efficient Resilient MPC Scheme via Constraint Tightening against Cyberattacks: Application to Vehicle Cruise Control

We propose a novel framework for designing a resilient Model Predictive Control (MPC) targeting uncertain linear systems under cyber attack. Assuming a periodic attack scenario, we model the system under Denial of Service (DoS) attack, also with measurement noise, as an uncertain linear system with parametric and additive uncertainty. To detect anomalies, we employ a Kalman filter-based approach. Then, through our observations of the intensity of the launched attack, we determine a range of possible values for the system matrices, as well as establish bounds of the additive uncertainty for the equivalent uncertain system. Leveraging a recent constraint tightening robust MPC method, we present an optimization-based resilient algorithm. Accordingly, we compute the uncertainty bounds and corresponding constraints offline for various attack magnitudes. Then, this data can be used efficiently in the MPC computations online. We demonstrate the effectiveness of the developed framework on the Adaptive Cruise Control (ACC) problem.Comment: To Appear in ICINCO 202

An improved Η∞ approach for networked control systems with transmission delays and packet dropout

This paper is concerned with the H infinity controller design for robust stabilization of networked control systems with the network-induced delay, data packet dropout and norm-bounded parameter uncertainties. In order to obtain less conservative results, a new augmented Lyapunov-Krasovskii functional is used and novel free-weighting matrices are employed to make some extra degree of freedom in the H infinity design conditions. The feedback gain of a memoryless controller, maximum allowable delay bound and minimum disturbance attenuation level can be derived by solving a set of linear matrix inequalities (LMIs). The advantages of the proposed method are demonstrated by numerical example

A less conservative criterion for stability analysis of linear systems with time-varying delay

A

SARS-CoV-2 vaccination modelling for safe surgery to save lives: data from an international prospective cohort study

Background: Preoperative SARS-CoV-2 vaccination could support safer elective surgery. Vaccine numbers are limited so this study aimed to inform their prioritization by modelling. Methods: The primary outcome was the number needed to vaccinate (NNV) to prevent one COVID-19-related death in 1 year. NNVs were based on postoperative SARS-CoV-2 rates and mortality in an international cohort study (surgical patients), and community SARS-CoV-2 incidence and case fatality data (general population). NNV estimates were stratified by age (18-49, 50-69, 70 or more years) and type of surgery. Best- and worst-case scenarios were used to describe uncertainty. Results: NNVs were more favourable in surgical patients than the general population. The most favourable NNVs were in patients aged 70 years or more needing cancer surgery (351; best case 196, worst case 816) or non-cancer surgery (733; best case 407, worst case 1664). Both exceeded the NNV in the general population (1840; best case 1196, worst case 3066). NNVs for surgical patients remained favourable at a range of SARS-CoV-2 incidence rates in sensitivity analysis modelling. Globally, prioritizing preoperative vaccination of patients needing elective surgery ahead of the general population could prevent an additional 58 687 (best case 115 007, worst case 20 177) COVID-19-related deaths in 1 year. Conclusion: As global roll out of SARS-CoV-2 vaccination proceeds, patients needing elective surgery should be prioritized ahead of the general population