A comparison of sample-based Stochastic Optimal Control methods

In this paper, we compare the performance of two scenario-based numerical methods to solve stochastic optimal control problems: scenario trees and particles. The problem consists in finding strategies to control a dynamical system perturbed by exogenous noises so as to minimize some expected cost along a discrete and finite time horizon. We introduce the Mean Squared Error (MSE) which is the expected $L^2$-distance between the strategy given by the algorithm and the optimal strategy, as a performance indicator for the two models. We study the behaviour of the MSE with respect to the number of scenarios used for discretization. The first model, widely studied in the Stochastic Programming community, consists in approximating the noise diffusion using a scenario tree representation. On a numerical example, we observe that the number of scenarios needed to obtain a given precision grows exponentially with the time horizon. In that sense, our conclusion on scenario trees is equivalent to the one in the work by Shapiro (2006) and has been widely noticed by practitioners. However, in the second part, we show using the same example that, by mixing Stochastic Programming and Dynamic Programming ideas, the particle method described by Carpentier et al (2009) copes with this numerical difficulty: the number of scenarios needed to obtain a given precision now does not depend on the time horizon. Unfortunately, we also observe that serious obstacles still arise from the system state space dimension

A stochastic minimum principle and an adaptive pathwise algorithm for stochastic optimal control

We present a numerical method for finite-horizon stochastic optimal control models. We derive a stochastic minimum principle (SMP) and then develop a numerical method based on the direct solution of the SMP. The method combines Monte Carlo pathwise simulation and non-parametric interpolation methods. We present results from a standard linear quadratic control model, and a realistic case study that captures the stochastic dynamics of intermittent power generation in the context of optimal economic dispatch models.National Science Foundation (U.S.) (Grant 1128147)United States. Dept. of Energy. Office of Science (Biological and Environmental Research Program Grant DE-SC0005171)United States. Dept. of Energy. Office of Science (Biological and Environmental Research Program Grant DE-SC0003906

Particle Methods For Stochastic Optimal Control Problems

International audienceTo tackle the difficulties faced by both stochastic dynamic programming and scenario tree methods, we present some variational approach for numerical solution of stochastic optimal control problems. We consider two different interpretations of the control problem, an algebraic and a functional one from which we derive optimality conditions. An adaptative mesh discretization method will be used to propose a tractable solution algorithm. An application to a hydro-electric dam production management problem will be presented