Randomized and Relaxed Strategies in Continuous-Time Markov Decision Processes.

One of the goals of this article is to describe a wide class of control strategies, which includes the traditional relaxed strategies, as well as the so called randomized strategies which appeared earlier only in the framework of semi-Markov decision processes. If the objective is the total expected cost up to the accumulation of jumps, then without loss of generality one can consider only Markov relaxed strategies. Under a simple condition, the Markov randomized strategies are also sufficient. An example shows that the mentioned condition is important. Finally, without any conditions, the class of so called Poisson-related strategies is also sufficient in the optimization problems. All the results are applicable to the discounted model, they may be useful also for the case of long-run average cost. Read More: https://epubs.siam.org/doi/10.1137/15M101401

Realizable strategies in continuous-time Markov decision processes

For the Borel model of the continuous-time Markov decision process, we introduce a wide class of control strategies. In a particular case, such strategies transform to the standard relaxed strategies, intensively studied in the last decade. In another special case, if one restricts to another special subclass of the general strategies, the model transforms to the semi-Markov decision process. Further, we show that the relaxed strategies are not realizable. For the constrained optimal control problem with total expected costs, we describe the sufficient class of realizable strategies, the so-called Poisson-related strategies. Finally, we show that, for solving the formulated optimal control problems, one can use all the tools developed earlier for the classical discrete-time Markov decision processes

Constrained and Unconstrained Optimal Discounted Control of Piecewise Deterministic Markov Processes

International audienceThe main goal of this paper is to study the infinite-horizon expected discounted continuous-time optimal control problem of piecewise deterministic Markov processes with the control acting continuously on the jump intensity $\lambda$ and on the transition measure $Q$ of the process but not on the deterministic flow $\phi$. The contributions of the paper are for the unconstrained as well as the constrained cases. The set of admissible control strategies is assumed to be formed by policies, possibly randomized and depending on the history of the process, taking values in a set valued action space. For the unconstrained case we provide sufficient conditions based on the three local characteristics of the process $\phi$, $\lambda$, $Q$ and the semicontinuity properties of the set valued action space, to guarantee the existence and uniqueness of the integro-differential optimality equation (the so-called Bellman--Hamilton--Jacobi equation) as well as the existence of an optimal (and $\delta$-optimal, as well) deterministic stationary control strategy for the problem. For the constrained case we show that the values of the constrained control problem and an associated infinite dimensional linear programming (LP) problem are the same, and moreover we provide sufficient conditions for the solvability of the LP problem as well as for the existence of an optimal feasible randomized stationary control strategy for the constrained problem

Optimal impulse control of dynamical systems

Using the tools of the Markov decision processes, we justify the dynamic programming approach to the optimal impulse control of deterministic dynamical systems. We prove the equivalence of the integral and differential forms of the optimality equation. The theory is illustrated by an example from mathematical epidemiology. The developed methods can be also useful for the study of piecewise deterministic Markov processes

Optimal policies for constrained average-cost Markov decision processes

Markov decision processes, Constraints, Stable measures, 90C40,

Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach

International audienc

Constrained Discounted Semimarkov Decision Processes

This paper reduces problems on the existence and the nding of optimal policies for multiple criterion discounted SMDPs to similar problems for MDPs. We prove this reduction and illustrate it by extending to SMDPs several results for constrained discounted MDPs