Verification of Uncertain POMDPs Using Barrier Certificates

We consider a class of partially observable Markov decision processes (POMDPs) with uncertain transition and/or observation probabilities. The uncertainty takes the form of probability intervals. Such uncertain POMDPs can be used, for example, to model autonomous agents with sensors with limited accuracy, or agents undergoing a sudden component failure, or structural damage [1]. Given an uncertain POMDP representation of the autonomous agent, our goal is to propose a method for checking whether the system will satisfy an optimal performance, while not violating a safety requirement (e.g. fuel level, velocity, and etc.). To this end, we cast the POMDP problem into a switched system scenario. We then take advantage of this switched system characterization and propose a method based on barrier certificates for optimality and/or safety verification. We then show that the verification task can be carried out computationally by sum-of-squares programming. We illustrate the efficacy of our method by applying it to a Mars rover exploration example.Comment: 8 pages, 4 figure

Safe Policy Synthesis in Multi-Agent POMDPs via Discrete-Time Barrier Functions

A multi-agent partially observable Markov decision process (MPOMDP) is a modeling paradigm used for high-level planning of heterogeneous autonomous agents subject to uncertainty and partial observation. Despite their modeling efficiency, MPOMDPs have not received significant attention in safety-critical settings. In this paper, we use barrier functions to design policies for MPOMDPs that ensure safety. Notably, our method does not rely on discretization of the belief space, or finite memory. To this end, we formulate sufficient and necessary conditions for the safety of a given set based on discrete-time barrier functions (DTBFs) and we demonstrate that our formulation also allows for Boolean compositions of DTBFs for representing more complicated safe sets. We show that the proposed method can be implemented online by a sequence of one-step greedy algorithms as a standalone safe controller or as a safety-filter given a nominal planning policy. We illustrate the efficiency of the proposed methodology based on DTBFs using a high-fidelity simulation of heterogeneous robots.Comment: 8 pages and 4 figure

Safe Policy Synthesis in Multi-Agent POMDPs via Discrete-Time Barrier Functions

A multi-agent partially observable Markov decision process (MPOMDP) is a modeling paradigm used for high-level planning of heterogeneous autonomous agents subject to uncertainty and partial observation. Despite their modeling efficiency, MPOMDPs have not received significant attention in safety-critical settings. In this paper, we use barrier functions to design policies for MPOMDPs that ensure safety. Notably, our method does not rely on discretizations of the belief space, or finite memory. To this end, we formulate sufficient and necessary conditions for the safety of a given set based on discrete-time barrier functions (DTBFs) and we demonstrate that our formulation also allows for Boolean compositions of DTBFs for representing more complicated safe sets. We show that the proposed method can be implemented online by a sequence of one-step greedy algorithms as a standalone safe controller or as a safety-filter given a nominal planning policy. We illustrate the efficiency of the proposed methodology based on DTBFs using a high-fidelity simulation of heterogeneous robots

Control Theory Meets POMDPs: A Hybrid Systems Approach

Partially observable Markov decision processes(POMDPs) provide a modeling framework for a variety of sequential decision making under uncertainty scenarios in artificial intelligence (AI). Since the states are not directly observable ina POMDP, decision making has to be performed based on the output of a Bayesian filter (continuous beliefs); hence, making POMDPs intractable to solve and analyze. To overcome the complexity challenge of POMDPs, we apply techniques from control theory. Our contributions are fourfold: (i) We begin by casting the problem of analyzing a POMDP into analyzing the behavior of a discrete-time switched system. Then, (ii) in order to estimate the reachable belief space of a POMDP, i.e., the set of all possible evolutions given an initial belief distribution over the states and a set of actions and observations, we find over-approximations in terms of sub-level sets of Lyapunov-like functions. Furthermore, (iii) in order to verify safety and performance requirements of a given POMDP, we formulate a barrier certificate theorem

Formal synthesis of partially-observable cyber-physical systems

This dissertation is motivated by the challenges arising in the synthesis of controllers for partially-observable cyber-physical systems (PO-CPSs). In the past decade, CPSs have become ubiquitous and an integral part of our daily lives. Examples of such systems range from autonomous vehicles, drones, and aircraft to robots and advanced manufacturing. In many applications, these systems are expected to do complex logic tasks. Such tasks can usually be expressed using temporal logic formulae or as (in)finite strings over finite automata. In the past few years, abstraction-based techniques have been very promising for the formal synthesis of controllers. Since these techniques are based on the discretization of state and input sets, when dealing with large-scale systems, unfortunately, they suffer severely from the curse of dimensionality (i.e., the computational complexity grows exponentially with the dimension of the state set). In order to overcome the large computa- tional burden, a discretization-free approach based on control barrier functions has shown great potential to solve formal synthesis problems. In this thesis, we provide a systematic approach to synthesize a hybrid control policy for partially-observable (stochastic) control systems without discretizing the state sets. In many real-life applications, full-state information is not always available (due to the cost of sensing or the unavailability of the measurements). Therefore, in this thesis, we consider partially-observable (stochastic) control systems. Given proper state estimators, our goal is to utilize a notion of control barrier functions to synthesize control policies that provide (and potentially maximize) a lower bound on the probability that the trajectories of the partially-observable (stochastic) control system satisfy complex logic specifications such as safety and those that can be expressed as deterministic finite automata (DFA). Two main approaches are presented in this thesis to construct control barrier functions. In the first approach, no prior knowledge of estimation accuracy is needed. The second approach utilizes a (probability) bound on the estimation accuracy. Though the synthesis procedure for lower-dimensional systems is challenging itself, the task is much more computationally expensive (if not impossible) for large-scale interconnected systems. To overcome the challenges encountered with large-scale systems, we develop approaches to reduce the computational complexity. In particular, by considering a large-scale partially-observable control system as an interconnection of lower-dimensional subsystems, we compute so-called local control barrier functions for subsystems along with the corresponding local controllers. By assuming some small-gain type conditions, we then utilize local control barrier functions of subsystems to compositionally construct an overall control barrier function for the interconnected system. Finally, since closed-form mathematical models of many physical systems are either unavailable or too complicated to be of any use, we also extend our work to the synthesis of safety controllers for partially-observable systems with unknown dynamics. To tackle this problem, we utilize a data-driven approach and construct control barrier functions and their corresponding controllers via sets of data collected from the output trajectories of the systems and the trajectories of the estimators. To demonstrate the effectiveness of the proposed results in the thesis, we consider various case studies, such as a DC motor, an adaptive cruise control (ACC) system consisting of vehicles in a platoon, and a Moore-Greitzer jet engine model

Safe Policy Synthesis in Multi-Agent POMDPs via Discrete-Time Barrier Functions

A multi-agent partially observable Markov decision process (MPOMDP) is a modeling paradigm used for high-level planning of heterogeneous autonomous agents subject to uncertainty and partial observation. Despite their modeling efficiency, MPOMDPs have not received significant attention in safety-critical settings. In this paper, we use barrier functions to design policies for MPOMDPs that ensure safety. Notably, our method does not rely on discretizations of the belief space, or finite memory. To this end, we formulate sufficient and necessary conditions for the safety of a given set based on discrete-time barrier functions (DTBFs) and we demonstrate that our formulation also allows for Boolean compositions of DTBFs for representing more complicated safe sets. We show that the proposed method can be implemented online by a sequence of one-step greedy algorithms as a standalone safe controller or as a safety-filter given a nominal planning policy. We illustrate the efficiency of the proposed methodology based on DTBFs using a high-fidelity simulation of heterogeneous robots