Exponential Convergence and stability of Howards's Policy Improvement Algorithm for Controlled Diffusions

Optimal control problems are inherently hard to solve as the optimization must be performed simultaneously with updating the underlying system. Starting from an initial guess, Howard's policy improvement algorithm separates the step of updating the trajectory of the dynamical system from the optimization and iterations of this should converge to the optimal control. In the discrete space-time setting this is often the case and even rates of convergence are known. In the continuous space-time setting of controlled diffusion the algorithm consists of solving a linear PDE followed by maximization problem. This has been shown to converge, in some situations, however no global rate of is known. The first main contribution of this paper is to establish global rate of convergence for the policy improvement algorithm and a variant, called here the gradient iteration algorithm. The second main contribution is the proof of stability of the algorithms under perturbations to both the accuracy of the linear PDE solution and the accuracy of the maximization step. The proof technique is new in this context as it uses the theory of backward stochastic differential equations.Comment: Identical to the published version except minor typographical detail

Time discretization of FBSDE with polynomial growth drivers and reaction-diffusion PDEs

In this paper, we undertake the error analysis of the time discretization of systems of Forward-Backward Stochastic Differential Equations (FBSDEs) with drivers having polynomial growth and that are also monotone in the state variable. We show with a counter-example that the natural explicit Euler scheme may diverge, unlike in the canonical Lipschitz driver case. This is due to the lack of a certain stability property of the Euler scheme which is essential to obtain convergence. However, a thorough analysis of the family of $\theta$-schemes reveals that this required stability property can be recovered if the scheme is sufficiently implicit. As a by-product of our analysis, we shed some light on higher order approximation schemes for FBSDEs under non-Lipschitz condition. We then return to fully explicit schemes and show that an appropriately tamed version of the explicit Euler scheme enjoys the required stability property and as a consequence converges. In order to establish convergence of the several discretizations, we extend the canonical path- and first-order variational regularity results to FBSDEs with polynomial growth drivers which are also monotone. These results are of independent interest for the theory of FBSDEs.Comment: Published at http://dx.doi.org/10.1214/14-AAP1056 in the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org

Almost sure exponential stability of numerical solutions for stochastic delay differential equations

Using techniques based on the continuous and discrete semimartingale convergence theorems, this paper investigates if numerical methods may reproduce the almost sure exponential stability of the exact solutions to stochastic delay differential equations (SDDEs). The important feature of this technique is that it enables us to study the almost sure exponential stability of numerical solutions of SDDEs directly. This is significantly different from most traditional methods by which the almost sure exponential stability is derived from the moment stability by the Chebyshev inequality and the Borel–Cantelli lemma