Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking

For exhaustive formal verification, industrial-scale cyber-physical systems (CPSs) are often too large and complex, and lightweight alternatives (e.g., monitoring and testing) have attracted the attention of both industrial practitioners and academic researchers. Falsification is one popular testing method of CPSs utilizing stochastic optimization. In state-of-the-art falsification methods, the result of the previous falsification trials is discarded, and we always try to falsify without any prior knowledge. To concisely memorize such prior information on the CPS model and exploit it, we employ Black-box checking (BBC), which is a combination of automata learning and model checking. Moreover, we enhance BBC using the robust semantics of STL formulas, which is the essential gadget in falsification. Our experiment results suggest that our robustness-guided BBC outperforms a state-of-the-art falsification tool.Comment: Accepted to HSCC 202

Time-Staging Enhancement of Hybrid System Falsification

Optimization-based falsification employs stochastic optimization algorithms to search for error input of hybrid systems. In this paper we introduce a simple idea to enhance falsification, namely time staging, that allows the time-causal structure of time-dependent signals to be exploited by the optimizers. Time staging consists of running a falsification solver multiple times, from one interval to another, incrementally constructing an input signal candidate. Our experiments show that time staging can dramatically increase performance in some realistic examples. We also present theoretical results that suggest the kinds of models and specifications for which time staging is likely to be effective

Huasheng Falsification Document

This document is part of a digital collection provided by the Martin P. Catherwood Library, ILR School, Cornell University, pertaining to the effects of globalization on the workplace worldwide.  Special emphasis is placed on labor rights, working conditions, labor market changes, and union organizing.CLW_Huasheng_Audit_Falsification.pdf: 32 downloads, before Oct. 1, 2020

A Falsification View of Success Typing

Dynamic languages are praised for their flexibility and expressiveness, but static analysis often yields many false positives and verification is cumbersome for lack of structure. Hence, unit testing is the prevalent incomplete method for validating programs in such languages. Falsification is an alternative approach that uncovers definite errors in programs. A falsifier computes a set of inputs that definitely crash a program. Success typing is a type-based approach to document programs in dynamic languages. We demonstrate that success typing is, in fact, an instance of falsification by mapping success (input) types into suitable logic formulae. Output types are represented by recursive types. We prove the correctness of our mapping (which establishes that success typing is falsification) and we report some experiences with a prototype implementation.Comment: extended versio

Falsification and refutation

A scientific theory, according to Popper, can be legitimately saved from falsification by introducing an auxiliary hypothesis to generate new, falsifiable predictions. Also, if there are suspicions of bias or error, the researchers might introduce an auxiliary falsifiable hypothesis that would allow testing. But this technique can not solve the problem in general, because any auxiliary hypothesis can be challenged in the same way, ad infinitum. To solve this regression, Popper introduces the idea of ​​a basic statement, an empirical statement that can be used both to determine whether a given theory is falsifiable and, if necessary, to corroborate falsification assumptions. DOI: 10.13140/RG.2.2.22162.0992

Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI

We demonstrate a unified approach to rigorous design of safety-critical autonomous systems using the VerifAI toolkit for formal analysis of AI-based systems. VerifAI provides an integrated toolchain for tasks spanning the design process, including modeling, falsification, debugging, and ML component retraining. We evaluate all of these applications in an industrial case study on an experimental autonomous aircraft taxiing system developed by Boeing, which uses a neural network to track the centerline of a runway. We define runway scenarios using the Scenic probabilistic programming language, and use them to drive tests in the X-Plane flight simulator. We first perform falsification, automatically finding environment conditions causing the system to violate its specification by deviating significantly from the centerline (or even leaving the runway entirely). Next, we use counterexample analysis to identify distinct failure cases, and confirm their root causes with specialized testing. Finally, we use the results of falsification and debugging to retrain the network, eliminating several failure cases and improving the overall performance of the closed-loop system.Comment: Full version of a CAV 2020 pape

The Falsification of Nuclear Forces

We review our work on the statistical uncertainty analysis of the NN force. This is based on the Granada-2013 database where a statistically meaningful partial wave analysis comprising a total of 6713 np and pp published scattering data from 1950 till 2013 below pion production threshold has been made. We stress the necessary conditions required for a correct and self-consistent statistical interpretation of the discrepancies between theory and experiment which enable a subsequent statistical error propagation and correlation analysisComment: 4 pages. Conference Proceedings. 21st International Conference on Few-Body Problems in Physics (FB21