6,661 research outputs found
Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters
Markov decision processes (MDPs) are a popular model for performance analysis
and optimization of stochastic systems. The parameters of stochastic behavior
of MDPs are estimates from empirical observations of a system; their values are
not known precisely. Different types of MDPs with uncertain, imprecise or
bounded transition rates or probabilities and rewards exist in the literature.
Commonly, analysis of models with uncertainties amounts to searching for the
most robust policy which means that the goal is to generate a policy with the
greatest lower bound on performance (or, symmetrically, the lowest upper bound
on costs). However, hedging against an unlikely worst case may lead to losses
in other situations. In general, one is interested in policies that behave well
in all situations which results in a multi-objective view on decision making.
In this paper, we consider policies for the expected discounted reward
measure of MDPs with uncertain parameters. In particular, the approach is
defined for bounded-parameter MDPs (BMDPs) [8]. In this setting the worst, best
and average case performances of a policy are analyzed simultaneously, which
yields a multi-scenario multi-objective optimization problem. The paper
presents and evaluates approaches to compute the pure Pareto optimal policies
in the value vector space.Comment: 9 pages, 5 figures, preprint for VALUETOOLS 201
Probabilistic Bisimulations for PCTL Model Checking of Interval MDPs
Verification of PCTL properties of MDPs with convex uncertainties has been
investigated recently by Puggelli et al. However, model checking algorithms
typically suffer from state space explosion. In this paper, we address
probabilistic bisimulation to reduce the size of such an MDPs while preserving
PCTL properties it satisfies. We discuss different interpretations of
uncertainty in the models which are studied in the literature and that result
in two different definitions of bisimulations. We give algorithms to compute
the quotients of these bisimulations in time polynomial in the size of the
model and exponential in the uncertain branching. Finally, we show by a case
study that large models in practice can have small branching and that a
substantial state space reduction can be achieved by our approach.Comment: In Proceedings SynCoP 2014, arXiv:1403.784
Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes
Interval Markov decision processes (IMDPs) generalise classical MDPs by
having interval-valued transition probabilities. They provide a powerful
modelling tool for probabilistic systems with an additional variation or
uncertainty that prevents the knowledge of the exact transition probabilities.
In this paper, we consider the problem of multi-objective robust strategy
synthesis for interval MDPs, where the aim is to find a robust strategy that
guarantees the satisfaction of multiple properties at the same time in face of
the transition probability uncertainty. We first show that this problem is
PSPACE-hard. Then, we provide a value iteration-based decision algorithm to
approximate the Pareto set of achievable points. We finally demonstrate the
practical effectiveness of our proposed approaches by applying them on several
case studies using a prototypical tool.Comment: This article is a full version of a paper accepted to the Conference
on Quantitative Evaluation of SysTems (QEST) 201
On Markov Chains with Uncertain Data
In this paper, a general method is described to determine uncertainty intervals for performance measures of Markov chains given an uncertainty region for the parameters of the Markov chains. We investigate the effects of uncertainties in the transition probabilities on the limiting distributions, on the state probabilities after n steps, on mean sojourn times in transient states, and on absorption probabilities for absorbing states. We show that the uncertainty effects can be calculated by solving linear programming problems in the case of interval uncertainty for the transition probabilities, and by second order cone optimization in the case of ellipsoidal uncertainty. Many examples are given, especially Markovian queueing examples, to illustrate the theory.Markov chain;Interval uncertainty;Ellipsoidal uncertainty;Linear Programming;Second Order Cone Optimization
Hitting times and probabilities for imprecise Markov chains
We consider the problem of characterising expected hitting times and hitting probabilities for imprecise Markov chains. To this end, we consider three distinct ways in which imprecise Markov chains have been defined in the literature: as sets of homogeneous Markov chains, as sets of more general stochastic processes, and as game-theoretic probability models. Our first contribution is that all these different types of imprecise Markov chains have the same lower and upper expected hitting times, and similarly the hitting probabilities are the same for these three types. Moreover, we provide a characterisation of these quantities that directly generalises a similar characterisation for precise, homogeneous Markov chains
Trading Safety Versus Performance: Rapid Deployment of Robotic Swarms with Robust Performance Constraints
In this paper we consider a stochastic deployment problem, where a robotic
swarm is tasked with the objective of positioning at least one robot at each of
a set of pre-assigned targets while meeting a temporal deadline. Travel times
and failure rates are stochastic but related, inasmuch as failure rates
increase with speed. To maximize chances of success while meeting the deadline,
a control strategy has therefore to balance safety and performance. Our
approach is to cast the problem within the theory of constrained Markov
Decision Processes, whereby we seek to compute policies that maximize the
probability of successful deployment while ensuring that the expected duration
of the task is bounded by a given deadline. To account for uncertainties in the
problem parameters, we consider a robust formulation and we propose efficient
solution algorithms, which are of independent interest. Numerical experiments
confirming our theoretical results are presented and discussed
- …