On gradual-impulse control of continuous-time Markov decision processes with multiplicative cost

In this paper, we consider the gradual-impulse control problem of continuous-time Markov decision processes, where the system performance is measured by the expectation of the exponential utility of the total cost. We prove, under very general conditions on the system primitives, the existence of a deterministic stationary optimal policy out of a more general class of policies. Policies that we consider allow multiple simultaneous impulses, randomized selection of impulses with random effects, relaxed gradual controls, and accumulation of jumps. After characterizing the value function using the optimality equation, we reduce the continuous-time gradual-impulse control problem to an equivalent simple discrete-time Markov decision process, whose action space is the union of the sets of gradual and impulsive actions

Numerical method for impulse control of Piecewise Deterministic Markov Processes

This paper presents a numerical method to calculate the value function for a general discounted impulse control problem for piecewise deterministic Markov processes. Our approach is based on a quantization technique for the underlying Markov chain defined by the post jump location and inter-arrival time. Convergence results are obtained and more importantly we are able to give a convergence rate of the algorithm. The paper is illustrated by a numerical example.Comment: This work was supported by ARPEGE program of the French National Agency of Research (ANR), project "FAUTOCOES", number ANR-09-SEGI-00

Average Continuous Control of Piecewise Deterministic Markov Processes

This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. The control variable acts on the jump rate and transition measure of the PDMP, and the running and boundary costs are assumed to be positive but not necessarily bounded. Our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete-time optimality equation related to the embedded Markov chain given by the post-jump location of the PDMP. Our second main result guarantees the existence of a feedback measurable selector for the discrete-time optimality equation by establishing a connection between this equation and an integro-differential equation. Our final main result is to obtain some sufficient conditions for the existence of a solution for a discrete-time optimality inequality and an ordinary optimal feedback control for the long run average cost using the so-called vanishing discount approach.Comment: 34 page

Maintenance optimization with piecewise deterministic Markov processes

International audienc

Piecewise deterministic Markov process for condition-based imperfect maintenance models

In this paper, a condition-based imperfect maintenance model based on piecewise deterministic Markov process (PDMP) is constructed. The degradation of the system includes two types: natural degradation and random shocks. The natural degradation is deterministic and can be nonlinear. The damage increment caused by a random shock follows a certain distribution, and its parameters are related to the degradation state. Maintenance methods include corrective maintenance and imperfect maintenance. Imperfect maintenance reduces the degradation degree of the system according to a random proportion. The maintenance action is delayed, and the system will suffer natural degradations and random shocks while waiting for maintenance. At each inspection time, the decision-maker needs to make a choice among planning no maintenance, imperfect maintenance and perfect maintenance, so as to minimize the total discounted cost of the system. The impulse optimal control theory of PDMP is used to determine the optimal maintenance strategy. A numerical study dealing with component coating maintenance problem is presented. Relationship with optimal threshold strategy is discussed. Sensitivity analyses on the influences of discount factor, observation interval and maintenance cost to the discounted cost and optimal actions are presented.Comment: 34 pages, 28 figure