Solving Factored MDPs with Hybrid State and Action Variables

Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a new hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function by a linear combination of basis functions and optimize its weights by linear programming. We analyze both theoretical and computational aspects of this approach, and demonstrate its scale-up potential on several hybrid optimization problems

Solving dependability/performability irreducible Markov models using regenerative randomization

Markov models are commonly used to asses the dependability/performability of fault-tolerant systems. Computation of many dependability/performability measures for repairable fault-tolerant systems requires the transient analysis of irreducible Markov models. Examples of such measures are the unavailability at time t and the expected interval unavailability at time t. Randomization (also called uniformization) is a well-known Markov transient analysis method and has good properties: numerical stability, well-controlled computation error, and ability to specify the computation error in advance. However, the randomization method is computationally expensive when the model is stiff, as is the case for Markov models of repairable fault-tolerant systems when the mission time of interest is large. Steady-state detection is a technique recently proposed to speedup randomization when the model is irreducible. This paper points out that another method, regenerative randomization, which has the same good properties as randomization, also covers irreducible models, and compares, for the important class of irreducible failure/repair models with exponential failure and repair time distributions and repair in every state with failed components, the efficiency of the regenerative randomization method with that of randomization with steady-state detection. In the frequent case in which the initial state is the state without failed components the regenerative randomization method can be faster than randomization with steady-state detection, specially when the model is large and the failure rates are much smaller than the repair rates. For other initial probability distributions, the regenerative randomization method seems to perform worse than randomization with steady-state detection.Postprint (published version