An efficient policy iteration algorithm for dynamic programming equations

We present an accelerated algorithm for the solution of static Hamilton–Jacobi–Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. This limitation often degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a coarse-mesh value iteration phase and then switches to a fine-mesh policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computations with the value iteration and at the same time to ensure the convergence of the policy iteration method to the optimal solution. We analyze the methods and efficient coupling in a number of examples in different dimensions, illustrating their properties

Supervised learning for kinetic consensus control

In this paper, how to successfully and efficiently condition a target population of agents towards consensus is discussed. To overcome the curse of dimensionality, the mean field formulation of the consensus control problem is considered. Although such formulation is designed to be independent of the number of agents, it is feasible to solve only for moderate intrinsic dimensions of the agents space. For this reason, the solution is approached by means of a Boltzmann procedure, i.e. quasi-invariant limit of controlled binary interactions as approximation of the mean field PDE. The need for an efficient solver for the binary interaction control problem motivates the use of a supervised learning approach to encode a binary feedback map to be sampled at a very high rate. A gradient augmented feedforward neural network for the Value function of the binary control problem is considered and compared with direct approximation of the feedback law

Optimizing semilinear representations for State-dependent Riccati Equation-based feedback control

An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law stemming from the solution of the corresponding Hamilton Jacobi Bellman (HJB) equation. Numerical experiments assess effectiveness of the method in terms of stability of the closed-loop with near-to-optimal performance

Optimizing semilinear representations for State-dependent Riccati Equation-based feedback control

An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law stemming from the solution of the corresponding Hamilton Jacobi Bellman (HJB) equation. Numerical experiments assess effectiveness of the method in terms of stability of the closed-loop with near-to-optimal performance

Gradient-augmented supervised learning of optimal feedback laws using state-dependent Riccati equations

A supervised learning approach for the solution of large-scale nonlinear stabilization problems is presented. A stabilizing feedback law is trained from a dataset generated from State-dependent Riccati Equation solvers. The training phase is enriched by the use of gradient information in the loss function, which is weighted through the use of hyperparameters. High-dimensional nonlinear stabilization tests demonstrate that real-time sequential large-scale Algebraic Riccati Equation solvers can be substituted by a suitably trained feedforward neural network

State-dependent Riccati equation feedback stabilization for nonlinear PDEs

The synthesis of suboptimal feedback laws for controlling nonlinear dynamics arising from semi-discretized PDEs is studied. An approach based on the State-dependent Riccati Equation (SDRE) is presented for 2 and ∞ control problems. Depending on the nonlinearity and the dimension of the resulting problem, offline, online, and hybrid offline-online alternatives to the SDRE synthesis are proposed. The hybrid offline-online SDRE method reduces to the sequential solution of Lyapunov equations, effectively enabling the computation of suboptimal feedback controls for two-dimensional PDEs. Numerical tests for the Sine-Gordon, degenerate Zeldovich, and viscous Burgers’ PDEs are presented, providing a thorough experimental assessment of the proposed methodology

Optimal actuator design for the Euler-Bernoulli vibration model based on LQR performance and shape calculus

A method for optimal actuator design in vibration control is presented. The optimal actuator, parametrized as a characteristic function, is found by means of the topological derivative of the LQR cost. An abstract framework is proposed based on the theory for infinite-dimensional optimization of both the actuator shape and the associated control problem. A numerical realization of the optimality condition is developed for the actuator shape using a level-set method for topological derivatives. A numerical example illustrating the design of actuator for Euler-Bernoulli beam model is provided

Using mobility data in the design of optimal lockdown strategies for the COVID-19 pandemic

A mathematical model for the COVID-19 pandemic spread, which integratesage-structured Susceptible-Exposed-Infected-Recovered-Deceased dynamics with realmobile phone data accounting for the population mobility, is presented. The dynamicalmodel adjustment is performed via Approximate Bayesian Computation. Optimallockdown and exit strategies are determined based on nonlinear model predictivecontrol, constrained to public-health and socio-economic factors. Through an extensivecomputational validation of the methodology, it is shown that it is possible to computerobust exit strategies with realistic reduced mobility values to inform public policymaking, and we exemplify the applicability of the methodology using datasets fromEngland and France

Proximal methods for stationary mean field games with local couplings

© 2018 Society for Industrial and Applied Mathematics. We address the numerical approximation of mean field games with local couplings. For power-like Hamiltonians, we consider a stationary system and also a system involving density constraints modeling hard congestion effects. For finite difference discretization of the mean field game system developed in [Y. Achdou and I. Capuzzo-Dolcetta, SIAM J. Numer. Anal., 48 (2010), pp. 1136-1162], we follow a variational approach. We prove that the aforementioned schemes can be obtained as the optimality system of suitably defined optimization problems. In order to prove the existence of solutions of the scheme with a variational argument, monotonicity assumptions on the coupling term are not needed, which allows us to recover general existence results proved by Achdou and Capuzzo-Dolcetta. Next, assuming that the coupling term is nondecreasing, the variational problem is cast as a convex optimization problem, for which we study and compare several proximal-type methods. These algorithms have several interesting features, such as global convergence and stability with respect to the viscosity parameter, which can eventually be zero. We assess the performance of the methods via numerical experiments

Data-driven initialization of deep learning solvers for Hamilton-Jacobi-Bellman PDEs

A deep learning approach for the approximation of the Hamilton-Jacobi-Bellman partial differential equation (HJB PDE) associated to the Nonlinear Quadratic Regulator (NLQR) problem. A state-dependent Riccati equation control law is first used to generate a gradient-augmented synthetic dataset for supervised learning. The resulting model becomes a warm start for the minimization of a loss function based on the residual of the HJB PDE. The combination of supervised learning and residual minimization avoids spurious solutions and mitigate the data inefficiency of a supervised learning-only approach. Numerical tests validate the different advantages of the proposed methodology