17,491 research outputs found
One-counter Markov decision processes
We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs are equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a stochastic model studied heavily in queueing theory and applied probability. They can thus be viewed as a natural ``adversarial'' version of a classic stochastic model. Alternatively, they can also be viewed as a natural probabilistic/controlled extension of classic one-counter automata. OC-MDPs also subsume (as a very restricted special case) a recently studied MDP model called ``solvency games'' that model a risk-averse gambling scenario. Basic computational questions about these models include ``termination'' questions and ``limit'' questions, such as the following: does the controller have a ``strategy'' (or ``policy'') to ensure that the counter (which may for example count the number of jobs in the queue) will hit value 0 (the empty queue) almost surely (a.s.)? Or that it will have infinite limsup value, a.s.? Or, that it will hit value 0 in selected terminal states, a.s.? Or, in case these are not satisfied a.s., compute the maximum (supremum) such probability over all strategies. We provide new upper and lower bounds on the complexity of such problems. For some of them we present a polynomial-time algorithm, whereas for others we show PSPACE- or BH-hardness and give an EXPTIME upper bound. Our upper bounds combine techniques from the theory of MDP reward models, the theory of random walks, and a variety of automata-theoretic methods
One-Counter Stochastic Games
We study the computational complexity of basic decision problems for
one-counter simple stochastic games (OC-SSGs), under various objectives.
OC-SSGs are 2-player turn-based stochastic games played on the transition graph
of classic one-counter automata. We study primarily the termination objective,
where the goal of one player is to maximize the probability of reaching counter
value 0, while the other player wishes to avoid this. Partly motivated by the
goal of understanding termination objectives, we also study certain "limit" and
"long run average" reward objectives that are closely related to some
well-studied objectives for stochastic games with rewards. Examples of problems
we address include: does player 1 have a strategy to ensure that the counter
eventually hits 0, i.e., terminates, almost surely, regardless of what player 2
does? Or that the liminf (or limsup) counter value equals infinity with a
desired probability? Or that the long run average reward is >0 with desired
probability? We show that the qualitative termination problem for OC-SSGs is in
NP intersection coNP, and is in P-time for 1-player OC-SSGs, or equivalently
for one-counter Markov Decision Processes (OC-MDPs). Moreover, we show that
quantitative limit problems for OC-SSGs are in NP intersection coNP, and are in
P-time for 1-player OC-MDPs. Both qualitative limit problems and qualitative
termination problems for OC-SSGs are already at least as hard as Condon's
quantitative decision problem for finite-state SSGs.Comment: 20 pages, 1 figure. This is a full version of a paper accepted for
publication in proceedings of FSTTCS 201
Analysis of Probabilistic Basic Parallel Processes
Basic Parallel Processes (BPPs) are a well-known subclass of Petri Nets. They
are the simplest common model of concurrent programs that allows unbounded
spawning of processes. In the probabilistic version of BPPs, every process
generates other processes according to a probability distribution. We study the
decidability and complexity of fundamental qualitative problems over
probabilistic BPPs -- in particular reachability with probability 1 of
different classes of target sets (e.g. upward-closed sets). Our results concern
both the Markov-chain model, where processes are scheduled randomly, and the
MDP model, where processes are picked by a scheduler.Comment: This is the technical report for a FoSSaCS'14 pape
Verification problems for timed and probabilistic extensions of Petri Nets
In the first part of the thesis, we prove the decidability (and PSPACE-completeness) of
the universal safety property on a timed extension of Petri Nets, called Timed Petri Nets.
Every token has a real-valued clock (a.k.a. age), and transition firing is constrained by
the clock values that have integer bounds (using strict and non-strict inequalities). The
newly created tokens can either inherit the age from an input token of the transition or
it can be reset to zero.
In the second part of the thesis, we refer to systems with controlled behaviour that
are probabilistic extensions of VASS and One-Counter Automata. Firstly, we consider
infinite state Markov Decision Processes (MDPs) that are induced by probabilistic
extensions of VASS, called VASS-MDPs. We show that most of the qualitative problems
for general VASS-MDPs are undecidable, and consider a monotone subclass in which
only the controller can change the counter values, called 1-VASS-MDPs. In particular,
we show that limit-sure control state reachability for 1-VASS-MDPs is decidable, i.e.,
checking whether one can reach a set of control states with probability arbitrarily close
to 1. Unlike for finite state MDPs, the control state reachability property may hold limit
surely (i.e. using an infinite family of strategies, each of which achieving the objective
with probability ≥ 1-e, for every e > 0), but not almost surely (i.e. with probability 1).
Secondly, we consider infinite state MDPs that are induced by probabilistic extensions of
One-Counter Automata, called One-Counter Markov Decision Processes (OC-MDPs).
We show that the almost-sure {1;2;3}-Parity problem for OC-MDPs is at least as hard
as the limit-sure selective termination problem for OC-MDPs, in which one would
like to reach a particular set of control states and counter value zero with probability
arbitrarily close to 1
- …