6,027 research outputs found
An Inductive Synthesis Framework for Verifiable Reinforcement Learning
Despite the tremendous advances that have been made in the last decade on
developing useful machine-learning applications, their wider adoption has been
hindered by the lack of strong assurance guarantees that can be made about
their behavior. In this paper, we consider how formal verification techniques
developed for traditional software systems can be repurposed for verification
of reinforcement learning-enabled ones, a particularly important class of
machine learning systems. Rather than enforcing safety by examining and
altering the structure of a complex neural network implementation, our
technique uses blackbox methods to synthesizes deterministic programs, simpler,
more interpretable, approximations of the network that can nonetheless
guarantee desired safety properties are preserved, even when the network is
deployed in unanticipated or previously unobserved environments. Our
methodology frames the problem of neural network verification in terms of a
counterexample and syntax-guided inductive synthesis procedure over these
programs. The synthesis procedure searches for both a deterministic program and
an inductive invariant over an infinite state transition system that represents
a specification of an application's control logic. Additional specifications
defining environment-based constraints can also be provided to further refine
the search space. Synthesized programs deployed in conjunction with a neural
network implementation dynamically enforce safety conditions by monitoring and
preventing potentially unsafe actions proposed by neural policies. Experimental
results over a wide range of cyber-physical applications demonstrate that
software-inspired formal verification techniques can be used to realize
trustworthy reinforcement learning systems with low overhead.Comment: Published on PLDI 201
Control and game-theoretic methods for secure cyber-physical-human systems
This work focuses on systems comprising tightly interconnected physical and digital components. Those, aptly named, cyber-physical systems will be the core of the Fourth Industrial Revolution. Thus, cyber-physical systems will be called upon to interact with humans, either in a cooperative fashion, or as adversaries to malicious human agents that will seek to corrupt their operation. In this work, we will present methods that enable an autonomous system to operate safely among human agents and to gain an advantage in cyber-physical security scenarios by employing tools from control, game and learning theories. Our work revolves around three main axes: unpredictability-based defense, operation among agents with bounded rationality and verification of safety properties for autonomous systems. In taking advantage of the complex nature of cyber-physical systems, our unpredictability-based defense work will focus both on attacks on actuating and sensing components, which will be addressed via a novel switching-based Moving Target Defense framework, and on Denial-of-Service attacks on the underlying network via a zero-sum game exploiting redundant communication channels. Subsequently, we will take a more abstract view of complex system security by exploring the principles of bounded rationality. We will show how attackers of bounded rationality can coordinate in inducing erroneous decisions to a system while they remain stealthy. Methods of cognitive hierarchy will be employed for decision prediction, while closed form solutions of the optimization problem and the conditions of convergence to the Nash equilibrium will be investigated. The principles of bounded rationality will be brought to control systems via the use of policy iteration algorithms, enabling data-driven attack prediction in a more realistic fashion than what can be offered by game equilibrium solutions. The issue of intelligence in security scenarios will be further considered via concepts of learning manipulation through a proposed framework where bounded rationality is understood as a hierarchy in learning, rather than optimizing, capability. This viewpoint will allow us to propose methods of exploiting the learning process of an imperfect opponent in order to affect their cognitive state via the use of tools from optimal control theory. Finally, in the context of safety, we will explore verification and compositionality properties of linear systems that are designed to be added to a cascade network of similar systems. To obfuscate the need for knowledge of the system's dynamics, we will state decentralized conditions that guarantee a specific dissipativity properties for the system, which are shown to be solved by reinforcement learning techniques. Subsequently, we will propose a framework that employs a hierarchical solution of temporal logic specifications and reinforcement learning problems for optimal tracking.Ph.D
Recommended from our members
Oracle-Guided Design and Analysis of Learning-Based Cyber-Physical Systems
We are in world where autonomous systems, such as self-driving cars, surgical robots, robotic manipulators are becoming a reality. Such systems are considered \textit{safety-critical} since they interact with humans on a regular basis. Hence, before such systems can be integrated into our day to day life, we need to guarantee their safety. Recent success in machine learning (ML) and artificial intelligence (AI) has led to an increase in their use in real world robotic systems. For example, complex perception modules in self-driving cars and deep reinforcement learning controllers in robotic manipulators. Although powerful, they introduce an additional level of complexity when it comes to the formal analysis of autonomous systems. In this thesis, such systems are designated as Learning-Based Cyber-Physical Systems~(LB-CPS). In this thesis, we take inspiration from the Oracle-Guided Inductive Synthesis~(OGIS) paradigm to develop frameworks which can aid in achieving formal guarantees in different stages of an autonomous system design and analysis pipeline. Furthermore, we show that to guarantee the safety of LB-CPS, the design (synthesis) and analysis (verification) must consider feedback from the other. We consider five important parts of the design and analysis process and show a strong coupling among them, namely (i) Robust Control Synthesis from High Level Safety Specifications; (ii) Diagnosis and Repair of Safety Requirements for Control Synthesis; (iii) Counter-example Guided Data Augmentation for training high-accuracy ML models; (iv) Simulation-Guided Falsification and Verification against Adversarial Environments; and (v) Bridging Model and Real-World Gap. Finally, we introduce a software toolkit \verifai{} for the design and analysis of AI based systems, which was developed to provide a common formal platform to implement design and analysis frameworks for LB-CPS
Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking
For exhaustive formal verification, industrial-scale cyber-physical systems
(CPSs) are often too large and complex, and lightweight alternatives (e.g.,
monitoring and testing) have attracted the attention of both industrial
practitioners and academic researchers. Falsification is one popular testing
method of CPSs utilizing stochastic optimization. In state-of-the-art
falsification methods, the result of the previous falsification trials is
discarded, and we always try to falsify without any prior knowledge. To
concisely memorize such prior information on the CPS model and exploit it, we
employ Black-box checking (BBC), which is a combination of automata learning
and model checking. Moreover, we enhance BBC using the robust semantics of STL
formulas, which is the essential gadget in falsification. Our experiment
results suggest that our robustness-guided BBC outperforms a state-of-the-art
falsification tool.Comment: Accepted to HSCC 202
Compositional Verification for Autonomous Systems with Deep Learning Components
As autonomy becomes prevalent in many applications, ranging from
recommendation systems to fully autonomous vehicles, there is an increased need
to provide safety guarantees for such systems. The problem is difficult, as
these are large, complex systems which operate in uncertain environments,
requiring data-driven machine-learning components. However, learning techniques
such as Deep Neural Networks, widely used today, are inherently unpredictable
and lack the theoretical foundations to provide strong assurance guarantees. We
present a compositional approach for the scalable, formal verification of
autonomous systems that contain Deep Neural Network components. The approach
uses assume-guarantee reasoning whereby {\em contracts}, encoding the
input-output behavior of individual components, allow the designer to model and
incorporate the behavior of the learning-enabled components working
side-by-side with the other components. We illustrate the approach on an
example taken from the autonomous vehicles domain
Research Priorities for Robust and Beneficial Artificial Intelligence
Success in the quest for artificial intelligence has the potential to bring
unprecedented benefits to humanity, and it is therefore worthwhile to
investigate how to maximize these benefits while avoiding potential pitfalls.
This article gives numerous examples (which should by no means be construed as
an exhaustive list) of such worthwhile research aimed at ensuring that AI
remains robust and beneficial.Comment: This article gives examples of the type of research advocated by the
open letter for robust & beneficial AI at
http://futureoflife.org/ai-open-lette
- …