5 research outputs found
Symbolic Controller Synthesis for B\"uchi Specifications on Stochastic Systems
We consider the policy synthesis problem for continuous-state controlled
Markov processes evolving in discrete time, when the specification is given as
a B\"uchi condition (visit a set of states infinitely often). We decompose
computation of the maximal probability of satisfying the B\"uchi condition into
two steps. The first step is to compute the maximal qualitative winning set,
from where the B\"uchi condition can be enforced with probability one. The
second step is to find the maximal probability of reaching the already computed
qualitative winning set. In contrast with finite-state models, we show that
such a computation only gives a lower bound on the maximal probability where
the gap can be non-zero.
In this paper we focus on approximating the qualitative winning set, while
pointing out that the existing approaches for unbounded reachability
computation can solve the second step. We provide an abstraction-based
technique to approximate the qualitative winning set by simultaneously using an
over- and under-approximation of the probabilistic transition relation. Since
we are interested in qualitative properties, the abstraction is
non-probabilistic; instead, the probabilistic transitions are assumed to be
under the control of a (fair) adversary. Thus, we reduce the original policy
synthesis problem to a B\"uchi game under a fairness assumption and
characterize upper and lower bounds on winning sets as nested fixed point
expressions in the -calculus. This characterization immediately provides a
symbolic algorithm scheme. Further, a winning strategy computed on the abstract
game can be refined to a policy on the controlled Markov process.
We describe a concrete abstraction procedure and demonstrate our algorithm on
two case studies
The Linear Programming Approach to Reach-Avoid Problems for Markov Decision Processes
One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.ISSN:1076-975