An Efficient Policy Iteration Algorithm for Dynamic Programming Equations

We present an accelerated algorithm for the solution of static Hamilton-Jacobi-Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. In many cases, this limitation degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a value iteration phase and then switches to a policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computation with the value iteration and, at the same time, to be reasonably sure that the policy iteration method will finally converge to the optimal solution. We analyze the methods and efficient coupling in a number of examples in dimension two, three and four illustrating its properties

An efficient policy iteration algorithm for dynamic programming equations

We present an accelerated algorithm for the solution of static Hamilton–Jacobi–Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. This limitation often degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a coarse-mesh value iteration phase and then switches to a fine-mesh policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computations with the value iteration and at the same time to ensure the convergence of the policy iteration method to the optimal solution. We analyze the methods and efficient coupling in a number of examples in different dimensions, illustrating their properties

A HJB-POD approach for the control of nonlinear PDEs on a tree structure

The Dynamic Programming approach allows to compute a feedback control for nonlinear problems, but suffers from the curse of dimensionality. The computation of the control relies on the resolution of a nonlinear PDE, the Hamilton-Jacobi-Bellman equation, with the same dimension of the original problem. Recently, a new numerical method to compute the value function on a tree structure has been introduced. The method allows to work without a structured grid and avoids any interpolation. Here, we aim to test the algorithm for nonlinear two dimensional PDEs. We apply model order reduction to decrease the computational complexity since the tree structure algorithm requires to solve many PDEs. Furthermore, we prove an error estimate which guarantees the convergence of the proposed method. Finally, we show efficiency of the method through numerical tests

A Semi-Lagrangian scheme for a modified version of the Hughes model for pedestrian flow

In this paper we present a Semi-Lagrangian scheme for a regularized version of the Hughes model for pedestrian flow. Hughes originally proposed a coupled nonlinear PDE system describing the evolution of a large pedestrian group trying to exit a domain as fast as possible. The original model corresponds to a system of a conservation law for the pedestrian density and an Eikonal equation to determine the weighted distance to the exit. We consider this model in presence of small diffusion and discuss the numerical analysis of the proposed Semi-Lagrangian scheme. Furthermore we illustrate the effect of small diffusion on the exit time with various numerical experiments

A HJB-POD approach for the control of nonlinear PDEs on a tree structure

The Dynamic Programming approach allows to compute a feedback control for nonlinear problems, but suffers from the curse of dimensionality. The computation of the control relies on the resolution of a nonlinear PDE, the Hamilton-Jacobi-Bellman equation, with the same dimension of the original problem. Recently, a new numerical method to compute the value function on a tree structure has been introduced. The method allows to work without a structured grid and avoids any interpolation. Here, we aim at testing the algorithm for nonlinear two dimensional PDEs. We apply model order reduction to decrease the computational complexity since the tree structure algorithm requires to solve many PDEs. Furthermore, we prove an error estimate which guarantees the convergence of the proposed method. Finally, we show efficiency of the method through numerical tests

Optimal Raw Material Inventory Analysis Using Markov Decision Process with Policy Iteration Method

Inventory of raw materials is a big deal in every production process, both in company production and home business production. In order to meet consumer demand, a business must be able to determine the amount of inventory that should be provided. The purpose of this research is to choose an alternative selection of ordering raw materials that produce the maximum amount of raw materials with minimum costs. The raw material referred to in this study is pandan leaves used to make pandan mats. Analysis of raw material inventory used in this research was the Markov decision process with the policy iteration method by considering the discount factor. From the analysis conducted, it is obtained alternative policies that must be taken by producers to meet raw materials with minimum costs. The results of this study can be a consideration for business actors in the study location in deciding the optimal ordering policy that should be taken to obtain the minimum operational cost