68 research outputs found
PReMo : An Analyzer for P robabilistic Re cursive Mo dels
This paper describes PReMo, a tool for analyzing Recursive Markov Chains, and their controlled/game extensions: (1-exit) Recursive Markov Decision Processes and Recursive Simple Stochastic Games
The complexity of computing a (quasi-)perfect equilibrium for an <i>n</i>-player extensive form game
One-Counter Stochastic Games
We study the computational complexity of basic decision problems for one-counter simple stochastic games (OC-SSGs), under various objectives. OC-SSGs are 2-player turn-based stochastic games played on the transition graph of classic one-counter automata. We study primarily the termination objective, where the goal of one player is to maximize the probability of reaching counter value 0, while the other player wishes to avoid this. Partly motivated by the goal of understanding termination objectives, we also study certain ``limit\u27\u27 and ``long run average\u27\u27 reward objectives that are closely related to some well-studied objectives for stochastic games with rewards. Examples of problems we address include: does player 1 have a
strategy to ensure that the counter eventually hits 0, i.e., terminates, almost surely, regardless of what player 2 does? Or that the (or ) counter value equals with a desired
probability? Or that the long run average reward is with desired probability? We show that the qualitative termination problem
for OC-SSGs is in intersect , and is in P-time for 1-player OC-SSGs, or equivalently for one-counter Markov Decision Processes (OC-MDPs). Moreover, we show that quantitative limit problems for OC-SSGs are in intersect , and are in P-time for 1-player OC-MDPs. Both qualitative limit problems and qualitative termination problems for OC-SSGs are already at least as hard as Condon\u27s quantitative decision problem for finite-state SSGs
Polynomial Time Algorithms for Multi-Type Branching Processes and Stochastic Context-Free Grammars
We show that one can approximate the least fixed point solution for a
multivariate system of monotone probabilistic polynomial equations in time
polynomial in both the encoding size of the system of equations and in
log(1/\epsilon), where \epsilon > 0 is the desired additive error bound of the
solution. (The model of computation is the standard Turing machine model.)
We use this result to resolve several open problems regarding the
computational complexity of computing key quantities associated with some
classic and heavily studied stochastic processes, including multi-type
branching processes and stochastic context-free grammars
One-Counter Stochastic Games
We study the computational complexity of basic decision problems for
one-counter simple stochastic games (OC-SSGs), under various objectives.
OC-SSGs are 2-player turn-based stochastic games played on the transition graph
of classic one-counter automata. We study primarily the termination objective,
where the goal of one player is to maximize the probability of reaching counter
value 0, while the other player wishes to avoid this. Partly motivated by the
goal of understanding termination objectives, we also study certain "limit" and
"long run average" reward objectives that are closely related to some
well-studied objectives for stochastic games with rewards. Examples of problems
we address include: does player 1 have a strategy to ensure that the counter
eventually hits 0, i.e., terminates, almost surely, regardless of what player 2
does? Or that the liminf (or limsup) counter value equals infinity with a
desired probability? Or that the long run average reward is >0 with desired
probability? We show that the qualitative termination problem for OC-SSGs is in
NP intersection coNP, and is in P-time for 1-player OC-SSGs, or equivalently
for one-counter Markov Decision Processes (OC-MDPs). Moreover, we show that
quantitative limit problems for OC-SSGs are in NP intersection coNP, and are in
P-time for 1-player OC-MDPs. Both qualitative limit problems and qualitative
termination problems for OC-SSGs are already at least as hard as Condon's
quantitative decision problem for finite-state SSGs.Comment: 20 pages, 1 figure. This is a full version of a paper accepted for
publication in proceedings of FSTTCS 201
Recursive Concurrent Stochastic Games
We study Recursive Concurrent Stochastic Games (RCSGs), extending our recent
analysis of recursive simple stochastic games to a concurrent setting where the
two players choose moves simultaneously and independently at each state. For
multi-exit games, our earlier work already showed undecidability for basic
questions like termination, thus we focus on the important case of single-exit
RCSGs (1-RCSGs).
We first characterize the value of a 1-RCSG termination game as the least
fixed point solution of a system of nonlinear minimax functional equations, and
use it to show PSPACE decidability for the quantitative termination problem. We
then give a strategy improvement technique, which we use to show that player 1
(maximizer) has \epsilon-optimal randomized Stackless & Memoryless (r-SM)
strategies for all \epsilon > 0, while player 2 (minimizer) has optimal r-SM
strategies. Thus, such games are r-SM-determined. These results mirror and
generalize in a strong sense the randomized memoryless determinacy results for
finite stochastic games, and extend the classic Hoffman-Karp strategy
improvement approach from the finite to an infinite state setting. The proofs
in our infinite-state setting are very different however, relying on subtle
analytic properties of certain power series that arise from studying 1-RCSGs.
We show that our upper bounds, even for qualitative (probability 1)
termination, can not be improved, even to NP, without a major breakthrough, by
giving two reductions: first a P-time reduction from the long-standing
square-root sum problem to the quantitative termination decision problem for
finite concurrent stochastic games, and then a P-time reduction from the latter
problem to the qualitative termination problem for 1-RCSGs.Comment: 21 pages, 2 figure
The complexity of computing a (quasi-)perfect equilibrium for an n-player extensive form game of perfect recall.
We study the complexity of computing or approximating refinements of Nash
equilibrium for a given finite n-player extensive form game of perfect recall
(EFGPR), where n >= 3. Our results apply to a number of well-studied
refinements, including sequential (SE), extensive-form perfect (PE), and
quasi-perfect equilibrium (QPE). These refine Nash and subgame-perfect
equilibrium. Of these, the most refined notions are PE and QPE. By classic
results, all these equilibria exist in any EFGPR. We show that, for all these
notions of equilibrium, approximating an equilibrium for a given EFGPR, to
within a given desired precision, is FIXP_a-complete. We also consider the
complexity of corresponding "almost" equilibrium notions, and show that they
are PPAD-complete. In particular, we define "delta-almost
epsilon-(quasi-)perfect" equilibrium, and show computing one is PPAD-complete.
We show these notions refine "delta-almost subgame-perfect equilibrium" for
EFGPRs, which is PPAD-complete. Thus, approximating one such (delta-almost)
equilibrium for n-player EFGPRs, n >= 3, is P-time equivalent to approximating
a (delta-almost) NE for a normal form game (NFG) with 3 or more players. NFGs
are trivially encodable as EFGPRs without blowup in size. Thus our results
extend the celebrated complexity results for NFGs to refinements of equilibrium
in the more general setting of EFGPRs. For 2-player EFGPRs, analogous
complexity results follow from the algorithms of Koller, Megiddo, and von
Stengel (1996), von Stengel, van den Elzen, and Talman (2002), and Miltersen
and Soerensen (2010). For n-player EFGPRs, an analogous result for Nash and
subgame-perfect equilibrium was given by Daskalakis, Fabrikant, and
Papadimitriou (2006). However, no analogous results were known for the more
refined notions of equilibrium for EFGPRs with 3 or more players
The complexity of analyzing infinite-state Markov chains, Markov decision processes, and stochastic games (Invited talk)
In recent years, a considerable amount of research has been devoted to understanding the computational complexity of basic analysis problems, and model checking problems, for finitely-presented countable infinite-state probabilistic systems. In particular, we have studied recursive Markov chains (RMCs), recursive Markov decision processes (RMDPs) and recursive stochastic games (RSGs). These arise by adding a natural recursion feature to finite-state Markov chains, MDPs, and stochastic games. RMCs and RMDPs provide natural abstract models of probabilistic procedural programs with recursion, and they are expressively equivalent to probabilistic and MDP extensions of pushdown automata. Moreover, a number of well-studied stochastic processes, including multi-type branching processes, (discrete-time) quasi-birth-death processes, and stochastic context-free grammars, can be suitably captured by subclasses of RMCs.
A central computational problem for analyzing various classes of recursive probabilistic systems is the computation of their (optimal) termination probabilities. These form a key ingredient for many other analyses, including model checking. For RMCs, and for important subclasses of RMDPs and RSGs, computing their termination values is equivalent to computing the least fixed point (LFP) solution of a corresponding monotone system of polynomial (min/max) equations. The complexity of computing the LFP solution for such equation systems is a intriguing problem, with connections to several areas of research. The LFP solution may in general be irrational. So, one possible aim is to compute it to within a desired additive error epsilon > 0. For general RMCs, approximating their termination probability within any non-trivial constant additive error < 1/2, is at least as hard as long-standing open problems in the complexity of numerical computation which are not even known to be in NP. For several key subclasses of RMCs and RMDPs, computing their termination values
turns out to be much more tractable.
In this talk I will survey algorithms for, and discuss the computational complexity of, key analysis problems for classes of infinite-state recursive MCs, MDPs, and stochastic games. In particular, I will discuss recent joint work with Alistair Stewart and Mihalis Yannakakis (in papers that appeared at STOC\u2712 and ICALP\u2712), in which we have obtained polynomial time algorithms for computing, to within arbitrary desired precision, the LFP solution of probabilistic polynomial (min/max) systems of equations. Using this, we obtained the first P-time algorithms for computing (within desired precision) the extinction probabilities of multi-type branching processes, the probability that an arbitrary given stochastic context-free grammar generates a given string, and the optimum (maximum or minimum) extinction probabilities for branching MDPs and context-free MDPs. For branching MDPs, their corresponding equations amount to Bellman optimality equations for minimizing/maximizing their termination probabilities. Our algorithms combine variations and generalizations of Newton\u27s method with other techniques, including linear programming. The algorithms are fairly easy to implement, but analyzing their worst-case running time
is mathematically quite involved
Multi-Objective Model Checking of Markov Decision Processes
We study and provide efficient algorithms for multi-objective model checking
problems for Markov Decision Processes (MDPs). Given an MDP, M, and given
multiple linear-time (\omega -regular or LTL) properties \varphi\_i, and
probabilities r\_i \epsilon [0,1], i=1,...,k, we ask whether there exists a
strategy \sigma for the controller such that, for all i, the probability that a
trajectory of M controlled by \sigma satisfies \varphi\_i is at least r\_i. We
provide an algorithm that decides whether there exists such a strategy and if
so produces it, and which runs in time polynomial in the size of the MDP. Such
a strategy may require the use of both randomization and memory. We also
consider more general multi-objective \omega -regular queries, which we
motivate with an application to assume-guarantee compositional reasoning for
probabilistic systems.
Note that there can be trade-offs between different properties: satisfying
property \varphi\_1 with high probability may necessitate satisfying \varphi\_2
with low probability. Viewing this as a multi-objective optimization problem,
we want information about the "trade-off curve" or Pareto curve for maximizing
the probabilities of different properties. We show that one can compute an
approximate Pareto curve with respect to a set of \omega -regular properties in
time polynomial in the size of the MDP.
Our quantitative upper bounds use LP methods. We also study qualitative
multi-objective model checking problems, and we show that these can be analysed
by purely graph-theoretic methods, even though the strategies may still require
both randomization and memory.Comment: 21 pages, 2 figure
- âŠ