Magnifying Lens Abstraction for Stochastic Games with Discounted and Long-run Average Objectives

Turn-based stochastic games and its important subclass Markov decision processes (MDPs) provide models for systems with both probabilistic and nondeterministic behaviors. We consider turn-based stochastic games with two classical quantitative objectives: discounted-sum and long-run average objectives. The game models and the quantitative objectives are widely used in probabilistic verification, planning, optimal inventory control, network protocol and performance analysis. Games and MDPs that model realistic systems often have very large state spaces, and probabilistic abstraction techniques are necessary to handle the state-space explosion. The commonly used full-abstraction techniques do not yield space-savings for systems that have many states with similar value, but does not necessarily have similar transition structure. A semi-abstraction technique, namely Magnifying-lens abstractions (MLA), that clusters states based on value only, disregarding differences in their transition relation was proposed for qualitative objectives (reachability and safety objectives). In this paper we extend the MLA technique to solve stochastic games with discounted-sum and long-run average objectives. We present the MLA technique based abstraction-refinement algorithm for stochastic games and MDPs with discounted-sum objectives. For long-run average objectives, our solution works for all MDPs and a sub-class of stochastic games where every state has the same value

Optimal Controller Synthesis for Nonlinear Systems

Optimal controller synthesis is a challenging problem to solve. However, in many applications such as robotics, nonlinearity is unavoidable. Apart from optimality, correctness of the system behaviors with respect to system specifications such as stability and obstacle avoidance is vital for engineering applications. Many existing techniques consider either the optimality or the correctness of system behavior. Rarely, a tool exists that considers both. Furthermore, most existing optimal controller synthesis techniques are not scalable because they either require ad-hoc design or they suffer from the curse of dimensionality. This thesis aims to close these gaps by proposing optimal controller synthesis techniques for two classes of nonlinear systems: linearly solvable nonlinear systems and hybrid nonlinear systems. Linearly solvable systems have associated Hamilton- Jacobi-Bellman (HJB) equations that can be transformed from the original nonlinear partial differential equation (PDE) into a linear PDE through a logarithmic transformation. The first part of this thesis presets two methods to synthesize optimal controller for linearly solvable nonlinear systems. The first technique uses a hierarchy of sums-of-square programs to compute a sequence of suboptimal controllers that have non-increasing suboptimality for first exit and finite horizon problems. This technique is the first systematic approach to provide stability and suboptimal performance guarantees for stochastic nonlinear systems in one framework. The second technique uses the low rank tensor decomposition framework to solve the linear HJB equation for first exit, finite horizon, and infinite horizon problems. This technique scale linearly with dimensions, alleviating the curse of dimensionality and enabling us to solve the linear HJB equation for a quadcopter model that is a twelve-dimensional system on a personal laptop. A new algorithm is proposed for a key step in the controller synthesis algorithm to solve the ill-conditioning issue that arises in the original algorithm. A MATLAB toolbox that implements the algorithms is developed, and the performance of these algorithms is illustrated by a few engineering examples. Apart from stability, in many applications, more complex specifications such as obstacle avoidance, reachability, and surveillance are required. The second part of the thesis describes methods to synthesize optimal controllers for hybrid nonlinear systems with quantitative objectives (i.e., minimizing cost) and qualitative objectives (i.e., satisfying specifications). This thesis focuses on two types of qualitative objectives, regular objectives, and ω-regular objectives. Regular objectives capture bounded time behavior such as reachability, and &#969;-regular objectives capture long term behavior such as surveillance. For both types of objectives, an abstraction-refinement procedure that preserves the cost is developed. A two-player game is solved on the product of the abstract system and the given objectives to synthesize the suboptimal controller for the hybrid nonlinear system. By refining the abstract system, the algorithms are guaranteed to converge to the optimal cost and return the optimal controller if the original systems are robust with respect to the initial states and the optimal controller inputs. The proposed technique is the first abstraction-refinement based technique to combine both quantitative and qualitative objectives into one framework. A Python implementation of the algorithms are developed, and a few engineering examples are presented to illustrate the performance of these algorithms.</p

Tools and Algorithms for the Construction and Analysis of Systems

This open access two-volume set constitutes the proceedings of the 27th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2021, which was held during March 27 – April 1, 2021, as part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021. The conference was planned to take place in Luxembourg and changed to an online format due to the COVID-19 pandemic. The total of 41 full papers presented in the proceedings was carefully reviewed and selected from 141 submissions. The volume also contains 7 tool papers; 6 Tool Demo papers, 9 SV-Comp Competition Papers. The papers are organized in topical sections as follows: Part I: Game Theory; SMT Verification; Probabilities; Timed Systems; Neural Networks; Analysis of Network Communication. Part II: Verification Techniques (not SMT); Case Studies; Proof Generation/Validation; Tool Papers; Tool Demo Papers; SV-Comp Tool Competition Papers

Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes

Interval Markov decision processes (IMDPs) generalise classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. In this paper, we consider the problem of multi-objective robust strategy synthesis for interval MDPs, where the aim is to find a robust strategy that guarantees the satisfaction of multiple properties at the same time in face of the transition probability uncertainty. We first show that this problem is PSPACE-hard. Then, we provide a value iteration-based decision algorithm to approximate the Pareto set of achievable points. We finally demonstrate the practical effectiveness of our proposed approaches by applying them on several case studies using a prototypical tool.Comment: This article is a full version of a paper accepted to the Conference on Quantitative Evaluation of SysTems (QEST) 201

Verification and Control of Partially Observable Probabilistic Real-Time Systems

We propose automated techniques for the verification and control of probabilistic real-time systems that are only partially observable. To formally model such systems, we define an extension of probabilistic timed automata in which local states are partially visible to an observer or controller. We give a probabilistic temporal logic that can express a range of quantitative properties of these models, relating to the probability of an event's occurrence or the expected value of a reward measure. We then propose techniques to either verify that such a property holds or to synthesise a controller for the model which makes it true. Our approach is based on an integer discretisation of the model's dense-time behaviour and a grid-based abstraction of the uncountable belief space induced by partial observability. The latter is necessarily approximate since the underlying problem is undecidable, however we show how both lower and upper bounds on numerical results can be generated. We illustrate the effectiveness of the approach by implementing it in the PRISM model checker and applying it to several case studies, from the domains of computer security and task scheduling

Verifiable and Compositional Reinforcement Learning Systems

We propose a novel framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL sub-systems, each of which learns to accomplish a separate sub-task, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of sub-systems, and of the collection of low-level sub-systems themselves. By defining interfaces between the sub-systems, the framework enables automatic decompositons of task specifications, e.g., reach a target set of states with a probability of at least 0.95, into individual sub-task specifications, i.e. achieve the sub-system's exit conditions with at least some minimum probability, given that its entry conditions are met. This in turn allows for the independent training and testing of the sub-systems; if they each learn a policy satisfying the appropriate sub-task specification, then their composition is guaranteed to satisfy the overall task specification. Conversely, if the sub-task specifications cannot all be satisfied by the learned policies, we present a method, formulated as the problem of finding an optimal set of parameters in the pMDP, to automatically update the sub-task specifications to account for the observed shortcomings. The result is an iterative procedure for defining sub-task specifications, and for training the sub-systems to meet them. As an additional benefit, this procedure allows for particularly challenging or important components of an overall task to be determined automatically, and focused on, during training. Experimental results demonstrate the presented framework's novel capabilities