Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations

Computing optimal feedback controls for nonlinear systems generally requires solving Hamilton-Jacobi-Bellman (HJB) equations, which are notoriously difficult when the state dimension is large. Existing strategies for high-dimensional problems often rely on specific, restrictive problem structures, or are valid only locally around some nominal trajectory. In this paper, we propose a data-driven method to approximate semi-global solutions to HJB equations for general high-dimensional nonlinear systems and compute candidate optimal feedback controls in real-time. To accomplish this, we model solutions to HJB equations with neural networks (NNs) trained on data generated without discretizing the state space. Training is made more effective and data-efficient by leveraging the known physics of the problem and using the partially-trained NN to aid in adaptive data generation. We demonstrate the effectiveness of our method by learning solutions to HJB equations corresponding to the attitude control of a six-dimensional nonlinear rigid body, and nonlinear systems of dimension up to 30 arising from the stabilization of a Burgers'-type partial differential equation. The trained NNs are then used for real-time feedback control of these systems.Comment: Added section on validation error computation. Updated convergence test formula and associated result

Adaptive deep learning for high-dimensional hamilton-jacobi-bellman equations

The article of record as published may be found at http://dx.doi.org/10.1137/19M1288802Computing optimal feedback controls for nonlinear systems generally requires solving Hamilton--Jacobi--Bellman (HJB) equations, which are notoriously difficult when the state dimension is large. Existing strategies for high-dimensional problems often rely on specific, restrictive problem structures or are valid only locally around some nominal trajectory. In this paper, we propose a data-driven method to approximate semiglobal solutions to HJB equations for general high-dimensional nonlinear systems and compute candidate optimal feedback controls in real-time. To accomplish this, we model solutions to HJB equations with neural networks (NNs) trained on data generated without discretizing the state space. Training is made more effective and data-efficient by leveraging the known physics of the problem and using the partially trained NN to aid in adaptive data generation. We demonstrate the effectiveness of our method by learning solutions to HJB equations corresponding to the attitude control of a six-dimensional nonlinear rigid body and nonlinear systems of dimension up to 30 arising from the stabilization of a Burgers'-type partial differential equation. The trained NNs are then used for real-time feedback control of these systems.Defense Advanced Research Projects Agency (DARPA)The work of the first and second authors was partially supported with funding from the Defense Advanced Research Projects Agency (DARPA) grant FA8650-18-1-7842

Deep neural network approximations for the stable manifolds of the Hamilton-Jacobi equations

As the Riccati equation for control of linear systems, the Hamilton-Jacobi-Bellman (HJB) equations play a fundamental role for optimal control of nonlinear systems. For infinite-horizon optimal control, the stabilizing solution of HJB equation can be represented by the stable manifold of the associated Hamiltonian system. In this paper, we study the neural network (NN) semiglobal approximation of the stable manifold. The main contribution includes two aspects: firstly, from the mathematical point of view, we rigorously prove that if an approximation is sufficiently close to the exact stable manifold of the HJB equation, then the corresponding control derived from this approximation is near optimal. Secondly, we propose a deep learning method to approximate the stable manifolds, and then numerically compute optimal feedback controls. The algorithm is devised from geometric features of the stable manifold, and relies on adaptive data generation by finding trajectories randomly in the stable manifold. The trajectories are found by solving two-point boundary value problems (BVP) locally near the equilibrium and extending the local solution by initial value problems (IVP) for the associated Hamiltonian system. A number of samples are chosen on each trajectory. Some adaptive samples are selected near the points with large errors after the previous round of training. Our algorithm is causality-free basically, hence it has a potential to apply to various high-dimensional nonlinear systems. We illustrate the effectiveness of our method by stabilizing the Reaction Wheel Pendulums.Comment: The algorithm is modified. The main point is that the trajectories on stable manifold are found by a combination of two-point BVP near the equilibrium and initial value problem far away from the equilibrium. The algorithm becomes more effectiv

QRnet: optimal regulator design with LQR-augmented neural networks

In this paper we propose a new computational method for designing optimal regulators for high-dimensional nonlinear systems. The proposed approach leverages physics-informed machine learning to solve high-dimensional Hamilton-Jacobi-Bellman equations arising in optimal feedback control. Concretely, we augment linear quadratic regulators with neural networks to handle nonlinearities. We train the augmented models on data generated without discretizing the state space, enabling application to high-dimensional problems. We use the proposed method to design a candidate optimal regulator for an unstable Burgers' equation, and through this example, demonstrate improved robustness and accuracy compared to existing neural network formulations.Comment: Added IEEE accepted manuscript with copyright notic

Linear Hamilton Jacobi Bellman Equations in High Dimensions

The Hamilton Jacobi Bellman Equation (HJB) provides the globally optimal solution to large classes of control problems. Unfortunately, this generality comes at a price, the calculation of such solutions is typically intractible for systems with more than moderate state space size due to the curse of dimensionality. This work combines recent results in the structure of the HJB, and its reduction to a linear Partial Differential Equation (PDE), with methods based on low rank tensor representations, known as a separated representations, to address the curse of dimensionality. The result is an algorithm to solve optimal control problems which scales linearly with the number of states in a system, and is applicable to systems that are nonlinear with stochastic forcing in finite-horizon, average cost, and first-exit settings. The method is demonstrated on inverted pendulum, VTOL aircraft, and quadcopter models, with system dimension two, six, and twelve respectively.Comment: 8 pages. Accepted to CDC 201

Gradient-augmented supervised learning of optimal feedback laws using state-dependent Riccati equations

A supervised learning approach for the solution of large-scale nonlinear stabilization problems is presented. A stabilizing feedback law is trained from a dataset generated from State-dependent Riccati Equation solvers. The training phase is enriched by the use of gradient information in the loss function, which is weighted through the use of hyperparameters. High-dimensional nonlinear stabilization tests demonstrate that real-time sequential large-scale Algebraic Riccati Equation solvers can be substituted by a suitably trained feedforward neural network