236 research outputs found
Maximizing the Conditional Expected Reward for Reaching the Goal
The paper addresses the problem of computing maximal conditional expected
accumulated rewards until reaching a target state (briefly called maximal
conditional expectations) in finite-state Markov decision processes where the
condition is given as a reachability constraint. Conditional expectations of
this type can, e.g., stand for the maximal expected termination time of
probabilistic programs with non-determinism, under the condition that the
program eventually terminates, or for the worst-case expected penalty to be
paid, assuming that at least three deadlines are missed. The main results of
the paper are (i) a polynomial-time algorithm to check the finiteness of
maximal conditional expectations, (ii) PSPACE-completeness for the threshold
problem in acyclic Markov decision processes where the task is to check whether
the maximal conditional expectation exceeds a given threshold, (iii) a
pseudo-polynomial-time algorithm for the threshold problem in the general
(cyclic) case, and (iv) an exponential-time algorithm for computing the maximal
conditional expectation and an optimal scheduler.Comment: 103 pages, extended version with appendices of a paper accepted at
TACAS 201
Energy loss in perturbative QCD
We review the properties of energetic parton propagation in hot or cold QCD
matter, as obtained in recent works. Advances in understanding the energy loss
- collisional and radiative - are summarized, with emphasis on the latter: it
features very interesting properties which may help to detect the quark-gluon
plasma produced in heavy ion collisions. We describe two different theoretical
approaches, which lead to the same radiated gluon energy spectrum. The case of
a longitudinally expanding QCD plasma is investigated. The energy lost by a jet
with given opening angle is calculated in view of making predictions for the
suppression (quenching) of hard jet production. Phenomenological implications
for the difference between hot and cold matter are discussed. Numerical
estimates of the loss suggest that it may be significantly enhanced in hot
compared to cold matter.Comment: 49 pages latex file with 11 embedded PS figures. Uses ar.sty
(included), one equation revised. submitted to Annual Review of Nuclear and
Particle Scienc
The Complexity of Graph-Based Reductions for Reachability in Markov Decision Processes
We study the never-worse relation (NWR) for Markov decision processes with an
infinite-horizon reachability objective. A state q is never worse than a state
p if the maximal probability of reaching the target set of states from p is at
most the same value from q, regard- less of the probabilities labelling the
transitions. Extremal-probability states, end components, and essential states
are all special cases of the equivalence relation induced by the NWR. Using the
NWR, states in the same equivalence class can be collapsed. Then, actions
leading to sub- optimal states can be removed. We show the natural decision
problem associated to computing the NWR is coNP-complete. Finally, we ex- tend
a previously known incomplete polynomial-time iterative algorithm to
under-approximate the NWR
Distributed Synthesis in Continuous Time
We introduce a formalism modelling communication of distributed agents
strictly in continuous-time. Within this framework, we study the problem of
synthesising local strategies for individual agents such that a specified set
of goal states is reached, or reached with at least a given probability. The
flow of time is modelled explicitly based on continuous-time randomness, with
two natural implications: First, the non-determinism stemming from interleaving
disappears. Second, when we restrict to a subclass of non-urgent models, the
quantitative value problem for two players can be solved in EXPTIME. Indeed,
the explicit continuous time enables players to communicate their states by
delaying synchronisation (which is unrestricted for non-urgent models). In
general, the problems are undecidable already for two players in the
quantitative case and three players in the qualitative case. The qualitative
undecidability is shown by a reduction to decentralized POMDPs for which we
provide the strongest (and rather surprising) undecidability result so far
Value Iteration for Long-run Average Reward in Markov Decision Processes
Markov decision processes (MDPs) are standard models for probabilistic
systems with non-deterministic behaviours. Long-run average rewards provide a
mathematically elegant formalism for expressing long term performance. Value
iteration (VI) is one of the simplest and most efficient algorithmic approaches
to MDPs with other properties, such as reachability objectives. Unfortunately,
a naive extension of VI does not work for MDPs with long-run average rewards,
as there is no known stopping criterion. In this work our contributions are
threefold. (1) We refute a conjecture related to stopping criteria for MDPs
with long-run average rewards. (2) We present two practical algorithms for MDPs
with long-run average rewards based on VI. First, we show that a combination of
applying VI locally for each maximal end-component (MEC) and VI for
reachability objectives can provide approximation guarantees. Second, extending
the above approach with a simulation-guided on-demand variant of VI, we present
an anytime algorithm that is able to deal with very large models. (3) Finally,
we present experimental results showing that our methods significantly
outperform the standard approaches on several benchmarks
Optimizing Performance of Continuous-Time Stochastic Systems using Timeout Synthesis
We consider parametric version of fixed-delay continuous-time Markov chains
(or equivalently deterministic and stochastic Petri nets, DSPN) where
fixed-delay transitions are specified by parameters, rather than concrete
values. Our goal is to synthesize values of these parameters that, for a given
cost function, minimise expected total cost incurred before reaching a given
set of target states. We show that under mild assumptions, optimal values of
parameters can be effectively approximated using translation to a Markov
decision process (MDP) whose actions correspond to discretized values of these
parameters
Percentile Queries in Multi-Dimensional Markov Decision Processes
Markov decision processes (MDPs) with multi-dimensional weights are useful to
analyze systems with multiple objectives that may be conflicting and require
the analysis of trade-offs. We study the complexity of percentile queries in
such MDPs and give algorithms to synthesize strategies that enforce such
constraints. Given a multi-dimensional weighted MDP and a quantitative payoff
function , thresholds (one per dimension), and probability thresholds
, we show how to compute a single strategy to enforce that for all
dimensions , the probability of outcomes satisfying is at least . We consider classical quantitative payoffs from
the literature (sup, inf, lim sup, lim inf, mean-payoff, truncated sum,
discounted sum). Our work extends to the quantitative case the multi-objective
model checking problem studied by Etessami et al. in unweighted MDPs.Comment: Extended version of CAV 2015 pape
Explicit Model Checking of Very Large MDP using Partitioning and Secondary Storage
The applicability of model checking is hindered by the state space explosion
problem in combination with limited amounts of main memory. To extend its
reach, the large available capacities of secondary storage such as hard disks
can be exploited. Due to the specific performance characteristics of secondary
storage technologies, specialised algorithms are required. In this paper, we
present a technique to use secondary storage for probabilistic model checking
of Markov decision processes. It combines state space exploration based on
partitioning with a block-iterative variant of value iteration over the same
partitions for the analysis of probabilistic reachability and expected-reward
properties. A sparse matrix-like representation is used to store partitions on
secondary storage in a compact format. All file accesses are sequential, and
compression can be used without affecting runtime. The technique has been
implemented within the Modest Toolset. We evaluate its performance on several
benchmark models of up to 3.5 billion states. In the analysis of time-bounded
properties on real-time models, our method neutralises the state space
explosion induced by the time bound in its entirety.Comment: The final publication is available at Springer via
http://dx.doi.org/10.1007/978-3-319-24953-7_1
An iterative decision-making scheme for Markov decision processes and its application to self-adaptive systems
Software is often governed by and thus adapts to phenomena that occur at runtime. Unlike traditional decision problems, where a decision-making model is determined for reasoning, the adaptation logic of such software is concerned with empirical data and is subject to practical constraints. We present an Iterative Decision-Making Scheme (IDMS) that infers both point and interval estimates for the undetermined transition probabilities in a Markov Decision Process (MDP) based on sampled data, and iteratively computes a confidently optimal scheduler from a given finite subset of schedulers. The most important feature of IDMS is the flexibility for adjusting the criterion of confident optimality and the sample size within the iteration, leading to a tradeoff between accuracy, data usage and computational overhead. We apply IDMS to an existing self-adaptation framework Rainbow and conduct a case study using a Rainbow system to demonstrate the flexibility of IDMS
- …