Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes

Interval Markov decision processes (IMDPs) generalise classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. In this paper, we consider the problem of multi-objective robust strategy synthesis for interval MDPs, where the aim is to find a robust strategy that guarantees the satisfaction of multiple properties at the same time in face of the transition probability uncertainty. We first show that this problem is PSPACE-hard. Then, we provide a value iteration-based decision algorithm to approximate the Pareto set of achievable points. We finally demonstrate the practical effectiveness of our proposed approaches by applying them on several case studies using a prototypical tool.Comment: This article is a full version of a paper accepted to the Conference on Quantitative Evaluation of SysTems (QEST) 201

Automatic deployment of autonomous cars in a robotic urban-like environment

Abstract-We present a computational framework and experimental setup for deployment of autonomous cars in a miniature Robotic Urban-Like Environment (RULE). The specifications are given in rich, human-like language as temporal logic statements about roads, intersections, and parking spaces. We use transition systems to model the motion and sensing capabilities of the robots and the topology of the environment and use tools resembling model checking to generate robot control strategies and to verify the correctness of the solution. The experimental setup is based on Khepera III robots, which move autonomously on streets while observing traffic rules

Gaze-based Intention Anticipation over Driving Manoeuvres in Semi-Autonomous Vehicles

Anticipating a human collaborator's intention enables safe and efficient interaction between a human and an autonomous system. Specifically, in the context of semiautonomous driving, studies have revealed that correct and timely prediction of the driver's intention needs to be an essential part of Advanced Driver Assistance System (ADAS) design. To this end, we propose a framework that exploits drivers' time-series eye gaze and fixation patterns to anticipate their real-time intention over possible future manoeuvres, enabling a smart and collaborative ADAS that can aid drivers to overcome safety-critical situations. The method models human intention as the latent states of a hidden Markov model and uses probabilistic dynamic time warping distributions to capture the temporal characteristics of the observation patterns of the drivers. The method is evaluated on a data set of 124 experiments from 75 drivers collected in a safety-critical semi-autonomous driving scenario. The results illustrate the efficacy of the framework by correctly anticipating the drivers' intentions about 3 seconds beforehand with over 90% accuracy

Omega-Regular Objectives in Model-Free Reinforcement Learning

We provide the first solution for model-free reinforcement learning of ω-regular objectives for Markov decision processes (MDPs). We present a constructive reduction from the almost-sure satisfaction of ω-regular bjectives to an almost-sure reachability problem, and extend this technique to learning how to control an unknown model so that the chance of satisfying the objective is maximized. We compile ω-regular properties into limit-deterministic B¨uchi automata instead of the traditional Rabin automata; this choice sidesteps difficulties that have marred previous proposals. Our approach allows us to apply model-free, off-the-shelf reinforcement learning algorithms to compute optimal strategies from the observations of the MDP. We present an experimental evaluation of our technique on benchmark learning problems

Formal verification and control of discrete-time stochastic systems

Thesis (Ph.D.)--Boston University PLEASE NOTE: Boston University Libraries did not receive an Authorization To Manage form for this thesis or dissertation. It is therefore not openly accessible, though it may be available by request. If you are the author or principal advisor of this work and would like to request open access for it, please contact us at [email protected]. Thank you.This thesis establishes theoretical and computational frameworks for formal verification and control synthesis for discrete-time stochastic systems. Given a temporal logic specification, the system is analyzed to determine the probability that the specification is achieved, and an input law is automatically generated to maximize this probability. The approach consists of three main steps: constructing an abstraction of the stochastic system as a finite Markov model, mapping the given specification onto this abstraction, and finding a control policy to maximize the probability of satisfying the specification. The framework uses Probabilistic Computation Tree Logic (PCTL) as the specification language. The verification and synthesis algorithms are inspired by the field of probabilistic model checking. In abstraction, a method for the computation of the exact transition probability bounds between the regions of interest in the domain of the stochastic system is first developed. These bounds are then used to construct an Interval-valued Markov Chain (IMC) or a Bounded-parameter Markov Decision Process (BMDP) abstraction for the system. Then, a representative transition probability is used to construct an approximating Markov chain (MC) for the stochastic system. The exact bound of the approximation error and an explicit expression for its grovvth over time are derived. To achieve a desired error value, an adaptive refinement algorithm that takes advantage of the linear dynamics of the system is employed. To verify the properties of the continuous domain stochastic system against a finite-time PCTL specification, IMC and BMDP verification algorithms are designed. These algorithms have low computational complexity and are inspired by the MC model checking algorithms. The low computational complexity is achieved by over approximating the probabilities of satisfaction. To increase the precision of the method, two adaptive refinement procedures are proposed. Furthermore, a method of generating the control strategy that maximizes the probability of satisfaction of a PCTL specification for Markov Decision Processes (MDPs) is developed. Through a similar method, a formal synthesis framework is constructed for continuous domain stochastic systems by utilizing their BMDP abstractions. These methodologies are then applied in robotics applications as a means of automatically deploying a mobile robot subject to noisy sensors and actuators from PCTL specifications. This technique is demonstrated through simulation and experimental case studies of deployment of a robot in an indoor environment. The contributions of the thesis include verification and synthesis frameworks for discrete time stochastic linear systems, abstraction schemes for stochastic systems to MCs, IMCs, and BMDPs, model checking algorithms with low computational complexity for IMCs and BMDPs against finite-time PCTL formulas, synthesis algorithms for Markov Decision Processes (MDPs) from PCTL formulas, and a computational framework for automatic deployment of a mobile robot from PCTL specifications. The approaches were validated by simulations and experiments. The algorithms and techniques in this thesis help to make discrete-time stochastic systems a more useful and effective class of models for analysis and control of real world systems

Specification revision for Markov decision processes with optimal trade-off

Optimal control policy synthesis for probabilistic systems from high-level specifications is increasingly often studied. One major question that is commonly faced, however, is what to do when the optimal probability of achieving the specification is not satisfactory? We address this question by viewing the specification as a soft constraint and present a synthesis framework for MDPs that encodes and automates specification revision in a trade-off for higher probability. The method uses co-safe LTL as the specification language and quantifies the revisions to the specification according to userdefined proposition costs. The framework computes a control policy that optimizes the trade-off between the probability of satisfaction and the cost of specification revision. The key idea of the method is a rule for the composition of the MDP, the automaton representing the specification, and the proposition costs such that all possible specification revisions along with their costs and probabilities of satisfaction are captured in one structure. The problem is then reduced to multi-objective optimization on an MDP. The power of the method is illustrated though simulations of a complex robotic scenario

Specification revision for Markov decision processes with optimal trade-off

Optimal control policy synthesis for probabilistic systems from high-level specifications is increasingly often studied. One major question that is commonly faced, however, is what to do when the optimal probability of achieving the specification is not satisfactory? We address this question by viewing the specification as a soft constraint and present a synthesis framework for MDPs that encodes and automates specification revision in a trade-off for higher probability. The method uses co-safe LTL as the specification language and quantifies the revisions to the specification according to userdefined proposition costs. The framework computes a control policy that optimizes the trade-off between the probability of satisfaction and the cost of specification revision. The key idea of the method is a rule for the composition of the MDP, the automaton representing the specification, and the proposition costs such that all possible specification revisions along with their costs and probabilities of satisfaction are captured in one structure. The problem is then reduced to multi-objective optimization on an MDP. The power of the method is illustrated though simulations of a complex robotic scenario

Correct-by-construction advanced driver assistance systems based on a cognitive architecture

Research into safety in autonomous and semi-autonomous vehicles has, so far, largely been focused on testing and validation through simulation. Due to the fact that failure of these autonomous systems is potentially life-endangering, formal methods arise as a complementary approach. This paper studies the application of formal methods to the verification of a human driver model built using the cognitive architecture ACT-R, and to the design of correct-by- construction Advanced Driver Assistance Systems (ADAS). The novelty lies in the integration of ACT-R in the formal analysis and an abstraction technique that enables finite representation of a large dimensional, continuous system in the form of a Markov process. The situation considered is a multi-lane highway driving scenario and the interactions that arise. The efficacy of the method is illustrated in two case studies with various driving conditions

Control of Markov Decision Processes from PCTL specifications

Abstract — We address the problem of controlling a Markov Decision Process (MDP) such that the probability of satisfying a temporal logic specification over a set of properties associated to its states is maximized. We focus on specifications given as formulas of Probabilistic Computation Tree Logic (PCTL) and show that controllers can be synthesized by adapting existing PCTL model checking algorithms. We illustrate the approach by applying it to the automatic deployment of a mobile robot in an indoor-like environment with respect to a PCTL specification. I