On Decidability of Time-Bounded Reachability in CTMDPs

We consider the time-bounded reachability problem for continuous-time Markov decision processes. We show that the problem is decidable subject to Schanuel's conjecture. Our decision procedure relies on the structure of optimal policies and the conditional decidability (under Schanuel's conjecture) of the theory of reals extended with exponential and trigonometric functions over bounded domains. We further show that any unconditional decidability result would imply unconditional decidability of the bounded continuous Skolem problem, or equivalently, the problem of checking if an exponential polynomial has a non-tangential zero in a bounded interval. We note that the latter problems are also decidable subject to Schanuel's conjecture but finding unconditional decision procedures remain longstanding open problems

Bisimulations and Logical Characterizations on Continuous-time Markov Decision Processes

In this paper we study strong and weak bisimulation equivalences for continuous-time Markov decision processes (CTMDPs) and the logical characterizations of these relations with respect to the continuous-time stochastic logic (CSL). For strong bisimulation, it is well known that it is strictly finer than CSL equivalence. In this paper we propose strong and weak bisimulations for CTMDPs and show that for a subclass of CTMDPs, strong and weak bisimulations are both sound and complete with respect to the equivalences induced by CSL and the sub-logic of CSL without next operator respectively. We then consider a standard extension of CSL, and show that it and its sub-logic without X can be fully characterized by strong and weak bisimulations respectively over arbitrary CTMDPs.Comment: The conference version of this paper was published at VMCAI 201

On Skolem-Hardness and Saturation Points in Markov Decision Processes

The Skolem problem and the related Positivity problem for linear recurrence sequences are outstanding number-theoretic problems whose decidability has been open for many decades. In this paper, the inherent mathematical difficulty of a series of optimization problems on Markov decision processes (MDPs) is shown by a reduction from the Positivity problem to the associated decision problems which establishes that the problems are also at least as hard as the Skolem problem as an immediate consequence. The optimization problems under consideration are two non-classical variants of the stochastic shortest path problem (SSPP) in terms of expected partial or conditional accumulated weights, the optimization of the conditional value-at-risk for accumulated weights, and two problems addressing the long-run satisfaction of path properties, namely the optimization of long-run probabilities of regular co-safety properties and the model-checking problem of the logic frequency-LTL. To prove the Positivity- and hence Skolem-hardness for the latter two problems, a new auxiliary path measure, called weighted long-run frequency, is introduced and the Positivity-hardness of the corresponding decision problem is shown as an intermediate step. For the partial and conditional SSPP on MDPs with non-negative weights and for the optimization of long-run probabilities of constrained reachability properties (aU b), solutions are known that rely on the identification of a bound on the accumulated weight or the number of consecutive visits to certain sates, called a saturation point, from which on optimal schedulers behave memorylessly. In this paper, it is shown that also the optimization of the conditional value-at-risk for the classical SSPP and of weighted long-run frequencies on MDPs with non-negative weights can be solved in pseudo-polynomial time exploiting the existence of a saturation point. As a consequence, one obtains the decidability of the qualitative model-checking problem of a frequency-LTL formula that is not included in the fragments with known solutions

Towards efficient analysis of Markov automata

One of the most expressive formalisms to model concurrent systems is Markov automata. They serve as a semantics for many higher-level formalisms, such as generalised stochastic Petri nets and dynamic fault trees. Two of the most challenging problems for Markov automata to date are (i) the optimal time-bounded reachability probability and (ii) the optimal long-run average rewards. In this thesis, we aim at designing efficient sound techniques to analyse them. We approach the problem of time-bounded reachability from two different angles. First, we study the properties of the optimal solution and exploit this knowledge to construct an efficient algorithm that approximates the optimal values up to a guaranteed error bound. This algorithm is exhaustive, i. e. it computes values for each state of the Markov automaton. This may be a limitation for very large or even infinite Markov automata. To address this issue we design a second algorithm that approximates the optimal solution by only working with part of the total state-space. For the problem of long-run average rewards there exists a polynomial algorithm based on linear programming. Instead of chasing a better theoretical complexity bound we search for a practical solution based on an iterative approach. We design a value iteration algorithm that in our empirical evaluation turns out to scale several orders of magnitude better than the linear programming based approach.Markov-Automaten bilden einen der ausdrucksstärksten Formalismen um Nebenläufige Systeme zu modellieren. Sie werden benutzt um die Semantik vieler höherer Formalismen wie stochastischer Petri-Netze [Mar95, EHZ10] und Dynamic Fault Trees [DBB90] zu beschreiben. Die zwei herausfordernder Probleme im Bereich der Analyse großer Markov- Automaten sind (i) die zeitbeschränkten Erreichbarkeitwahrscheinlichkeit und (ii) optimale langfristige durchschnittliche Rewards. Diese Arbeit zielt auf das Design effizienter und korrekter Techniken um sie zu untersuchen. Das Problem der zeitbeschränkten Erreichbarkeitswahrscheinlichkeit gehen wir aus zwei verschiedenen Richtungen an: Zum einen studieren wir die Eigenschaften optimaler Lösungen und nutzen dieses Wissen um einen effizienten Approximationsalgorithmus zu bilden, der optimale Werte bis auf eine garantierte Fehlertoleranz berechnet. Dieser Algorithmus basiert darauf, Werte für jeden Zustand des Markov-Automaten zu berechnen. Dies kann die Anwendbarkeit für große oder gar unendliche Automaten einschränken. Um diese Problem zu lösen präsentieren wir einen zweiten Algorithmus, der die optimale Lösung approximiert, und dabei ausschließlich einen Teil des Zustandsraumes betrachtet. Für das Problem der optimalen langfristigen durchschnittlichen Rewards gibt es einen polynomiellen Algorithmus auf Basis linearer Programmierung. Anstelle eine bessere theoretische Komplexität anzustreben, konzentrieren wir uns darauf, eine praktische Lösung auf Basis eines iterativen Ansatzes zu finden. Wie entwickeln einen Werte-iterierenden Algorithmus der in unserer empirischen Evaluation um mehrere Größenordnungen besser als der auf linearer Programmierung basierende Ansatz skaliert

Formal methods for motion planning and control in dynamic and partially known environments

This thesis is motivated by time and safety critical applications involving the use of autonomous vehicles to accomplish complex tasks in dynamic and partially known environments. We use temporal logic to formally express such complex tasks. Temporal logic specifications generalize the classical notions of stability and reachability widely studied within the control and hybrid systems communities. Given a model describing the motion of a robotic system in an environment and a formal task specification, the aim is to automatically synthesize a control policy that guarantees the satisfaction of the specification. This thesis presents novel control synthesis algorithms to tackle the problem of motion planning from temporal logic specifications in uncertain environments. For each one of the planning and control synthesis problems addressed in this dissertation, the proposed algorithms are implemented, evaluated, and validated thought experiments and/or simulations. The first part of this thesis focuses on a mobile robot whose success is measured by the completion of temporal logic tasks within a given period of time. In addition to such time constraints, the planning algorithm must also deal with the uncertainty that arises from the changes in the robot's workspace during task execution. In particular, we consider a robot deployed in a partitioned environment subjected to structural changes such as doors that can open and close. The motion of the robot is modeled as a continuous time Markov decision process and the robot's mission is expressed as a Continuous Stochastic Logic (CSL) formula. A complete framework to find a control strategy that satisfies a specification given as a CSL formula is introduced. The second part of this thesis addresses the synthesis of controllers that guarantee the satisfaction of a task specification expressed as a syntactically co-safe Linear Temporal Logic (scLTL) formula. In this case, uncertainty is characterized by the partial knowledge of the robot's environment. Two scenarios are considered. First, a distributed team of robots required to satisfy the specification over a set of service requests occurring at the vertices of a known graph representing the environment is examined. Second, a single agent motion planning problem from the specification over a set of properties known to be satised at the vertices of the known graph environment is studied. In both cases, we exploit the existence of o-the-shelf model checking and runtime verification tools, the efficiency of graph search algorithms, and the efficacy of exploration techniques to solve the motion planning problem constrained by the absence of complete information about the environment. The final part of this thesis extends uncertainty beyond the absence of a complete knowledge of the environment described above by considering a robot equipped with a noisy sensing system. In particular, the robot is tasked with satisfying a scLTL specification over a set of regions of interest known to be present in the environment. In such a case, although the robot is able to measure the properties characterizing such regions of interest, precisely determining the identity of these regions is not feasible. A mixed observability Markov decision process is used to represent the robot's actuation and sensing models. The control synthesis problem from scLTL formulas is then formulated as a maximum probability reachability problem on this model. The integration of dynamic programming, formal methods, and frontier-based exploration tools allow us to derive an algorithm to solve such a reachability problem

The big-O problem for labelled markov chains and weighted automata

Given two weighted automata, we consider the problem of whether one is big-O of the other, i.e., if the weight of every finite word in the first is not greater than some constant multiple of the weight in the second. We show that the problem is undecidable, even for the instantiation of weighted automata as labelled Markov chains. Moreover, even when it is known that one weighted automaton is big-O of another, the problem of finding or approximating the associated constant is also undecidable. Our positive results show that the big-O problem is polynomial-time solvable for unambiguous automata, coNP-complete for unlabelled weighted automata (i.e., when the alphabet is a single character) and decidable, subject to Schanuel’s conjecture, when the language is bounded (i.e., a subset of w_1^* … w_m^* for some finite words w_1,… ,w_m). On labelled Markov chains, the problem can be restated as a ratio total variation distance, which, instead of finding the maximum difference between the probabilities of any two events, finds the maximum ratio between the probabilities of any two events. The problem is related to ε-differential privacy, for which the optimal constant of the big-O notation is exactly exp(ε)