Control of singularly perturbed hybrid stochastic systems

In this paper, we study a class of optimal stochastic control problems involving two different time scales. The fast mode of the system is represented by deterministic state equations whereas the slow mode of the system corresponds to a jump disturbance process. Under a fundamental “ergodicity” property for a class of “infinitesimal control systems” associated with the fast mode, we show that there exists a limit problem which provides a good approximation to the optimal control of the perturbed system. Both the finite- and infinite-discounted horizon cases are considered. We show how an approximate optimal control law can be constructed from the solution of the limit control problem. In the particular case where the infinitesimal control systems possess the so-called turnpike property, i.e., are characterized by the existence of global attractors, the limit control problem can be given an interpretation related to a decomposition approach

Decomposition and parallel processing techniques for two-time scale controlled Markov chains

This paper deals with a class of ergodic control problems for systems described by Markov chains with strong and weak interactions. These systems are composed of a set of m subchains that are weakly coupled. Using results recently established by Abbad et al. one formulates a limit control problem the solution of which can be obtained via an associated non-differentiable convex programming (NDCP) problem. The technique used to solve the NDCP problem is the Analytic Center Cutting Plane Method (ACCPM) which implements a dialogue between, on one hand, a master program computing the analytical center of a localization set containing the solution and, on the other hand, an oracle proposing cutting planes that reduce the size of the localization set at each main iteration. The interesting aspect of this implementation comes from two characteristics: (i) the oracle proposes cutting planes by solving reduced sized Markov Decision Problems (MDP) via a linear program (LP) or a policy iteration method; (ii) several cutting planes can be proposed simultaneously through a parallel implementation on m processors. The paper concentrates on these two aspects and shows, on a large scale MDP obtained from the numerical approximation "a la Kushner-Dupuis” of a singularly perturbed hybrid stochastic control problem, the important computational speed-up obtained

Averaging and linear programming in some singularly perturbed problems of optimal control

The paper aims at the development of an apparatus for analysis and construction of near optimal solutions of singularly perturbed (SP) optimal controls problems (that is, problems of optimal control of SP systems) considered on the infinite time horizon. We mostly focus on problems with time discounting criteria but a possibility of the extension of results to periodic optimization problems is discussed as well. Our consideration is based on earlier results on averaging of SP control systems and on linear programming formulations of optimal control problems. The idea that we exploit is to first asymptotically approximate a given problem of optimal control of the SP system by a certain averaged optimal control problem, then reformulate this averaged problem as an infinite-dimensional (ID) linear programming (LP) problem, and then approximate the latter by semi-infinite LP problems. We show that the optimal solution of these semi-infinite LP problems and their duals (that can be found with the help of a modification of an available LP software) allow one to construct near optimal controls of the SP system. We demonstrate the construction with two numerical examples.Comment: 53 pages, 10 figure

Multilevel algorithms for the optimization of structured problems

Although large scale optimization problems are very difficult to solve in general, problems that arise from practical applications often exhibit particular structure. In this thesis we study and improve algorithms that can efficiently solve structured problems. Three separate settings are considered. The first part concerns the topic of singularly perturbed Markov decision processes (MDPs). When a MDP is singularly perturbed, one can construct an aggregate model in which the solution is asymptotically optimal. We develop an algorithm that takes advantage of existing results to compute the solution of the original model. The proposed algorithm can compute the optimal solution with a reduction in complexity without any penalty in accuracy. In the second part, the class of empirical risk minimization (ERM) problems is studied. When using a first order method, the Lipschitz constant of the empirical risk plays a crucial role in the convergence analysis and stepsize strategy of these problems. We derive the probabilistic bounds for such Lipschitz constants using random matrix theory. Our results are used to derive the probabilistic complexity and develop a new stepsize strategy for first order methods. The proposed stepsize strategy, Probabilistic Upper-bound Guided stepsize strategy (PUG), has a strong theoretical guarantee on its performance compared to the standard stepsize strategy. In the third part, we extend the existing results on multilevel methods for unconstrained convex optimization. We study a special case where the hierarchy of models is created by approximating first and second order information of the exact model. This is known as Galerkin approximation, and we named the corresponding algorithm Galerkin-based Algebraic Multilevel Algorithm (GAMA). Three case studies are conducted to show how the structure of a problem could affect the convergence of GAMA.Open Acces

Control of singularly perturbed hybrid stochastic systems

In this paper we study a class of optimal stochastic control problems involving two different time scales. The fast mode of the system is represented by deterministic state equations whereas the slow mode of the system corresponds to a jump disturbance process. Under a fundamental ”ergodicity” property for a class of ”infinitesimal control systems” associated with the fast mode, we show that there exists a limit problem which provides a good approximation to the optimal control of the perturbed system. Both the finite and infinite discounted horizon cases are considered. We show how an approximate optimal control law can be constructed from the solution of the limit control problem. In the particular case where the infinitesimal control systems possess the so-called turnpike property, i.e. are characterized by the existence of global attractors, the limit control problem can be given an interpretation related to a decomposition approach

Analysis, estimation and control for perturbed and singular systems for systems subject to discrete events.

"The principle investigator for this effort is Professor Alan S. Willsky, and Professor George C. Verghese is co-principal investigator."--P. [3].Includes bibliographical references (p. [20]-[25]).Final technical report for grant AFOSR-88-0032.Supported by the AFOSR. AFOSR-88-003

International Conference on Dynamic Control and Optimization - DCO 2021: book of abstracts

Sem resumo disponível.publishe

LP Based Upper and Lower Bounds for Cesàro and Abel Limits of the Optimal Values in Problems of Control of Stochastic Discrete Time Systems

International audienceIn this paper, we study asymptotic properties of problems of control of stochastic discrete time systems (also known as Markov decision processes) with time averaging and time discounting optimality criteria, and we establish that the Cesàro and Abel limits of the optimal values in such problems can be evaluated with the help of a certain infinite-dimensional linear programming problem and its dual