An Efficient Policy Iteration Algorithm for Dynamic Programming Equations

We present an accelerated algorithm for the solution of static Hamilton-Jacobi-Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. In many cases, this limitation degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a value iteration phase and then switches to a policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computation with the value iteration and, at the same time, to be reasonably sure that the policy iteration method will finally converge to the optimal solution. We analyze the methods and efficient coupling in a number of examples in dimension two, three and four illustrating its properties

Robust feedback control of nonlinear PDEs by numerical approximation of high-dimensional Hamilton-Jacobi-Isaacs equations

Copyright © by SIAM. We propose an approach for the synthesis of robust and optimal feedback controllers for nonlinear PDEs. Our approach considers the approximation of infinite-dimensional control systems by a pseudospectral collocation method, leading to high-dimensional nonlinear dynamics. For the reducedorder model, we construct a robust feedback control based on the H∞ control method, which requires the solution of an associated high-dimensional Hamilton-Jacobi-Isaacs nonlinear PDE. The dimensionality of the Isaacs PDE is tackled by means of a separable representation of the control system, and a polynomial approximation ansatz for the corresponding value function. Our method proves to be effective for the robust stabilization of nonlinear dynamics up to dimension d ≈ 12. We assess the robustness and optimality features of our design over a class of nonlinear parabolic PDEs, including nonlinear advection and reaction terms. The proposed design yields a feedback controller achieving optimal stabilization and disturbance rejection properties, along with providing a modeling framework for the robust control of PDEs under parametric uncertainties

An efficient policy iteration algorithm for dynamic programming equations

We present an accelerated algorithm for the solution of static Hamilton–Jacobi–Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. This limitation often degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a coarse-mesh value iteration phase and then switches to a fine-mesh policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computations with the value iteration and at the same time to ensure the convergence of the policy iteration method to the optimal solution. We analyze the methods and efficient coupling in a number of examples in different dimensions, illustrating their properties

Value iteration convergence of ε-monotone schemes for stationary Hamilton-Jacobi equations

International audienceWe present an abstract convergence result for the xed point approximation of stationary Hamilton{Jacobi equations. The basic assumptions on the discrete operator are invariance with respect to the addition of constants, "-monotonicity and consistency. The result can be applied to various high-order approximation schemes which are illustrated in the paper. Several applications to Hamilton{Jacobi equations and numerical tests are presented

Boundary Treatment and Multigrid Preconditioning for Semi-Lagrangian Schemes Applied to Hamilton-Jacobi-Bellman Equations

We analyse two practical aspects that arise in the numerical solution of Hamilton-Jacobi-Bellman (HJB) equations by a particular class of monotone approximation schemes known as semi-Lagrangian schemes. These schemes make use of a wide stencil to achieve convergence and result in discretization matrices that are less sparse and less local than those coming from standard finite difference schemes. This leads to computational difficulties not encountered there. In particular, we consider the overstepping of the domain boundary and analyse the accuracy and stability of stencil truncation. This truncation imposes a stricter CFL condition for explicit schemes in the vicinity of boundaries than in the interior, such that implicit schemes become attractive. We then study the use of geometric, algebraic and aggregation-based multigrid preconditioners to solve the resulting discretised systems from implicit time stepping schemes efficiently. Finally, we illustrate the performance of these techniques numerically for benchmark test cases from the literature

Deep neural network approximations for the stable manifolds of the Hamilton-Jacobi equations

As the Riccati equation for control of linear systems, the Hamilton-Jacobi-Bellman (HJB) equations play a fundamental role for optimal control of nonlinear systems. For infinite-horizon optimal control, the stabilizing solution of HJB equation can be represented by the stable manifold of the associated Hamiltonian system. In this paper, we study the neural network (NN) semiglobal approximation of the stable manifold. The main contribution includes two aspects: firstly, from the mathematical point of view, we rigorously prove that if an approximation is sufficiently close to the exact stable manifold of the HJB equation, then the corresponding control derived from this approximation is near optimal. Secondly, we propose a deep learning method to approximate the stable manifolds, and then numerically compute optimal feedback controls. The algorithm is devised from geometric features of the stable manifold, and relies on adaptive data generation by finding trajectories randomly in the stable manifold. The trajectories are found by solving two-point boundary value problems (BVP) locally near the equilibrium and extending the local solution by initial value problems (IVP) for the associated Hamiltonian system. A number of samples are chosen on each trajectory. Some adaptive samples are selected near the points with large errors after the previous round of training. Our algorithm is causality-free basically, hence it has a potential to apply to various high-dimensional nonlinear systems. We illustrate the effectiveness of our method by stabilizing the Reaction Wheel Pendulums.Comment: The algorithm is modified. The main point is that the trajectories on stable manifold are found by a combination of two-point BVP near the equilibrium and initial value problem far away from the equilibrium. The algorithm becomes more effectiv