194 research outputs found
An Efficient Policy Iteration Algorithm for Dynamic Programming Equations
We present an accelerated algorithm for the solution of static
Hamilton-Jacobi-Bellman equations related to optimal control problems. Our
scheme is based on a classic policy iteration procedure, which is known to have
superlinear convergence in many relevant cases provided the initial guess is
sufficiently close to the solution. In many cases, this limitation degenerates
into a behavior similar to a value iteration method, with an increased
computation time. The new scheme circumvents this problem by combining the
advantages of both algorithms with an efficient coupling. The method starts
with a value iteration phase and then switches to a policy iteration procedure
when a certain error threshold is reached. A delicate point is to determine
this threshold in order to avoid cumbersome computation with the value
iteration and, at the same time, to be reasonably sure that the policy
iteration method will finally converge to the optimal solution. We analyze the
methods and efficient coupling in a number of examples in dimension two, three
and four illustrating its properties
Error estimates for a tree structure algorithm solving finite horizon control problems
In the Dynamic Programming approach to optimal control problems a crucial
role is played by the value function that is characterized as the unique
viscosity solution of a Hamilton-Jacobi-Bellman (HJB) equation. It is well
known that this approach suffers of the "curse of dimensionality" and this
limitation has reduced its practical in real world applications. Here we
analyze a dynamic programming algorithm based on a tree structure. The tree is
built by the time discrete dynamics avoiding in this way the use of a fixed
space grid which is the bottleneck for high-dimensional problems, this also
drops the projection on the grid in the approximation of the value function. We
present some error estimates for a first order approximation based on the
tree-structure algorithm. Moreover, we analyze a pruning technique for the tree
to reduce the complexity and minimize the computational effort. Finally, we
present some numerical tests
Polynomial approximation of high-dimensional Hamilton–Jacobi–Bellman equations and applications to feedback control of semilinear parabolic PDES
© 2018 Society for Industrial and Applied Mathematics. A procedure for the numerical approximation of high-dimensional Hamilton–Jacobi–Bellman (HJB) equations associated to optimal feedback control problems for semilinear parabolic equations is proposed. Its main ingredients are a pseudospectral collocation approximation of the PDE dynamics and an iterative method for the nonlinear HJB equation associated to the feedback synthesis. The latter is known as the successive Galerkin approximation. It can also be interpreted as Newton iteration for the HJB equation. At every step, the associated linear generalized HJB equation is approximated via a separable polynomial approximation ansatz. Stabilizing feedback controls are obtained from solutions to the HJB equations for systems of dimension up to fourteen
Mitigating the Curse of Dimensionality: Sparse Grid Characteristics Method for Optimal Feedback Control and HJB Equations
We address finding the semi-global solutions to optimal feedback control and
the Hamilton--Jacobi--Bellman (HJB) equation. Using the solution of an HJB
equation, a feedback optimal control law can be implemented in real-time with
minimum computational load. However, except for systems with two or three state
variables, using traditional techniques for numerically finding a semi-global
solution to an HJB equation for general nonlinear systems is infeasible due to
the curse of dimensionality. Here we present a new computational method for
finding feedback optimal control and solving HJB equations which is able to
mitigate the curse of dimensionality. We do not discretize the HJB equation
directly, instead we introduce a sparse grid in the state space and use the
Pontryagin's maximum principle to derive a set of necessary conditions in the
form of a boundary value problem, also known as the characteristic equations,
for each grid point. Using this approach, the method is spatially causality
free, which enjoys the advantage of perfect parallelism on a sparse grid.
Compared with dense grids, a sparse grid has a significantly reduced size which
is feasible for systems with relatively high dimensions, such as the -D
system shown in the examples. Once the solution obtained at each grid point,
high-order accurate polynomial interpolation is used to approximate the
feedback control at arbitrary points. We prove an upper bound for the
approximation error and approximate it numerically. This sparse grid
characteristics method is demonstrated with two examples of rigid body attitude
control using momentum wheels
Value iteration convergence of ε-monotone schemes for stationary Hamilton-Jacobi equations
International audienceWe present an abstract convergence result for the xed point approximation of stationary Hamilton{Jacobi equations. The basic assumptions on the discrete operator are invariance with respect to the addition of constants, "-monotonicity and consistency. The result can be applied to various high-order approximation schemes which are illustrated in the paper. Several applications to Hamilton{Jacobi equations and numerical tests are presented
Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations
Computing optimal feedback controls for nonlinear systems generally requires
solving Hamilton-Jacobi-Bellman (HJB) equations, which are notoriously
difficult when the state dimension is large. Existing strategies for
high-dimensional problems often rely on specific, restrictive problem
structures, or are valid only locally around some nominal trajectory. In this
paper, we propose a data-driven method to approximate semi-global solutions to
HJB equations for general high-dimensional nonlinear systems and compute
candidate optimal feedback controls in real-time. To accomplish this, we model
solutions to HJB equations with neural networks (NNs) trained on data generated
without discretizing the state space. Training is made more effective and
data-efficient by leveraging the known physics of the problem and using the
partially-trained NN to aid in adaptive data generation. We demonstrate the
effectiveness of our method by learning solutions to HJB equations
corresponding to the attitude control of a six-dimensional nonlinear rigid
body, and nonlinear systems of dimension up to 30 arising from the
stabilization of a Burgers'-type partial differential equation. The trained NNs
are then used for real-time feedback control of these systems.Comment: Added section on validation error computation. Updated convergence
test formula and associated result
- …